Building an Export Pipeline for CSV, JSON, and Webhooks

A "download as CSV" button looks simple in a mockup. Making it reliable for a survey with two million responses, without timing out the request or running the server out of memory, required a proper streaming export pipeline.

Streaming Instead of Buffering

Our first implementation loaded the full result set into memory before writing the file — fine for a thousand rows, catastrophic for a few million. We rewrote the export to stream rows directly from the database cursor to the output file, keeping memory usage constant regardless of export size.

Background Jobs for Large Exports

Exports above a size threshold run as background jobs rather than blocking an HTTP request. The customer receives an email with a download link once the export completes, rather than staring at a spinner that might eventually time out at the load balancer.

Format-Specific Considerations

CSV exports need careful handling of embedded commas, quotes, and newlines within free-text responses — we follow RFC 4180 quoting rules strictly rather than relying on a naive comma-join. JSON exports preserve full type fidelity (numbers stay numbers, nested answer objects stay nested) which CSV inherently can't represent.

Consistent Snapshots

An export that takes several minutes to generate against a live, constantly-updating dataset can produce an inconsistent snapshot if new responses arrive mid-export. We take a consistent read snapshot at the start of the export using the database's transaction isolation level, so the exported file always represents a single point in time.

Handling Partial Failures Gracefully

A multi-million-row export failing three-quarters of the way through, with no way to resume, wastes both compute time and the customer's patience. We checkpoint export progress periodically, so a transient database blip or worker restart resumes from the last checkpoint rather than starting over from zero — a small implementation detail that matters enormously for the largest exports on the platform.

Export pipelines are one of those features that seem finished after the happy-path demo but reveal their real complexity only once real customers push real data volume through them.