The limits of off-the-shelf ELT
The Cloudflare Business Intelligence team maintains a petabyte-scale data lake, ingesting thousands of tables daily from sources that range from internal databases like Postgres and ClickHouse to external SaaS applications such as Salesforce. The workloads are heavy — tables can collect hundreds of millions or billions of new rows per day — and the results drive product decisions, growth planning, and internal monitoring. In total, the pipeline moves about 141 billion rows every day.
As the data volume grew, the existing Extract Load Transform (ELT) approach stopped meeting requirements, and an evaluation of common alternatives showed none that would outperform it. The team concluded that the only path forward was to build an in-house framework tailored to their specific constraints. That framework is Jetflow.
Two orders of magnitude, one framework
The shift to Jetflow produced dramatic gains across several dimensions:
- Over 100x efficiency improvement in GB-s: The longest-running job, processing 19 billion rows, previously took 48 hours and consumed 300 GB of memory. It now completes in 5.5 hours using only 4 GB of memory.
- More than 10x performance improvement: The largest dataset previously ingested 60,000–80,000 rows per second; that figure is now 2–5 million rows per second per database connection, and scales further with multiple connections for certain databases.
- Extensibility: The modular design keeps it straightforward to extend and test. Jetflow currently supports ClickHouse, Postgres, Kafka, numerous SaaS APIs, Google BigQuery, and more, with new use cases continuing to fit without structural changes.
Requirements That Shaped the Design
Before writing any code, we established a clear set of requirements to guide the architecture. These were not just feature goals—they were constraints designed to prevent the framework from introducing new operational problems while solving existing ones.
Performance and Efficiency
Some of our ingestion jobs were taking roughly 24 hours, and our data volumes were only growing. The framework needed to move more data in less time, ingesting in a streaming fashion while consuming less memory and compute than our previous solution.
Backward Compatibility and Migration Path
With thousands of tables ingested daily, we needed the ability to migrate individual tables incrementally, not all at once. Because we rely on Spark downstream and it has limitations in merging disparate Parquet schemas, the framework had to generate precise schemas to match legacy output for each case. It also needed to integrate seamlessly with our custom metadata system for dependency checks and job status tracking.
Low Barrier to Entry
We wanted configuration files that could be version-controlled without creating bottlenecks in repositories with many concurrent changes. For most use cases, we required no-code or configuration-as-code, so users would not need to handle data type translation between source and target systems, or write new code for each ingestion. Schema should be inferred automatically from the source, keeping required configuration minimal.
Customizability When Needed
Balanced against the no-code requirement, we wanted an optional, flexible configuration layer for tuning and overrides. For instance, writing Parquet files is often costlier than reading from a database, so we needed to allocate more resources and concurrency per pipeline. We also wanted control over where work executes: workers could run in different threads, containers, or machines. The execution and data communication layer was abstracted behind an interface, with implementations injected via job configuration.
Testability
We needed a solution that runs locally in a containerized environment so we could test every stage of the pipeline. Black-box solutions force slow feedback loops where you validate output after a change without full visibility into code paths, making debugging painful and leaving edge cases untested.
Designing a Flexible Framework
The core idea was to decompose the pipeline into distinct, classified stages: Consumers, Transformers, and Loaders.
Pipelines are defined in a YAML file requiring one consumer, zero or more transformers, and at least one loader. Consumers create a data stream from the source system. Transformers—responsible for things like data transformation or validation—accept a stream and output a stream conforming to the same API so they can be chained. Loaders share that interface but are the stages with persistent effects, saving data to external systems. This modularity makes each stage independently testable, with shared behavior like error handling and concurrency inherited from base stages, cutting development time for new use cases.
Data Divisions for Idempotency
We designed a three-level data breakdown to ensure idempotency both on full pipeline re-runs and on internal retries of any data partition after transient errors, while still allowing parallel processing and targeted cleanup:
- RunInstance: the coarsest division, matching a business unit for a single run—for example, one month, day, or hour of data.
- Partition: a deterministic split of the RunInstance where each row maps to a partition based on self-evident attributes like account ID range or 10-minute intervals, without external state.
- Batch: a non-deterministic subdivision of partition data, used purely to break work into smaller chunks—like 10k rows or 50 MB—for streaming and parallel processing with fewer resources.
Consumer-stage YAML options construct the source query and also encode the semantic meaning of each data division in a system-agnostic way. Downstream stages therefore know what each partition represents—for instance, account IDs 0–500—enabling targeted cleanup and preventing duplicates when a single partition is retried.
Framework Implementation
Standard Internal State
Our most common pipeline reads from a database, converts to Parquet, and writes to object storage, with each step as a separate stage. To keep stages interoperable as new use cases onboarded, we took a strict approach: the extractor class only outputs data in a single internal format. Any downstream stage supporting that format as both input and output is compatible with the rest of the pipeline. This seems obvious now, but we initially built a custom type system and found it severely hindered stage interoperability.
We chose Arrow as that internal format for several reasons:
- Ecosystem adoption: Many data projects output Arrow, so writing extractors for new sources often means trivial Arrow production.
- Cheap data movement: Arrow moves between machines and languages with minimal serialization overhead. Since Jetflow was designed from the start for distributed execution via a job controller interface, efficient transmission preserves performance at scale.
- Reduced garbage collection pressure: Go’s GC cycle time is driven mainly by heap object count, not object size. Arrow allocates memory in large fixed-size batches. For 8192 rows with 10 columns, Arrow requires roughly 10 allocations versus the 8192 per-row allocations typical of most drivers.
Column-Oriented Processing
Most ingestion frameworks hold data as rows internally, but Parquet—our primary output—is columnar. When reading from column-based sources like ClickHouse, where drivers often return RowBinary format, converting to rows and back to columns is wasteful. Jetflow reads column-based sources directly in columnar formats—for example, ClickHouse’s native Block format—copies data into Arrow columns, and writes Parquet straight from Arrow. Eliminating those conversion steps provides a major performance boost.
Writing Pipeline Stages
ClickHouse Case Study
Testing an early Jetflow version against ClickHouse showed that adding more connections would not help: ClickHouse was returning data faster than we could receive it. The answer was to read more rows per second from a single connection using a better-optimized driver.
We initially wrote a custom ClickHouse driver but eventually switched to the ch-go low-level library, which reads Blocks directly in columnar form. Compared to the standard Go driver, performance improved dramatically. Combined with the framework optimizations, we now ingest millions of rows per second on a single ClickHouse connection. The takeaway: most database drivers optimize for the common case of row-at-a-time scanning, which has high per-row overhead and does not fit large-batch ingestion.
Postgres Case Study
For Postgres, we use the jackc/pgx driver but bypass the usual database/sql Scan interface. That standard interface relies on reflection to inspect and set typed fields for every column, which is too slow for us. Instead, we receive raw row bytes and call jackc/pgx’s internal scan functions per Postgres OID type. Since pgx reuses the row buffer between calls, this yields near-zero allocations per row, and we reach almost 600,000 rows per second per Postgres connection for most tables with very low memory usage.
Status and Next Steps
As of early July 2025, Jetflow ingests 77 billion records daily. Remaining jobs are migrating now, bringing the total to 141 billion records per day. The framework already handles tables we previously could not ingest, and runs with significantly less time and fewer resources. We plan to open source the project.
