R2 SQL: a deep dive into our new distributed query engine

A Serverless Query Engine for R2 Iceberg Tables

Running SQL over petabytes of object storage typically means standing up and managing a dedicated query cluster. R2 SQL takes a different path: a serverless engine that runs directly against Apache Iceberg tables in an R2 bucket, reading only the data a query actually needs and returning results in seconds.

The engine attacks the two fundamental obstacles to petabyte-scale querying with a two-phase design:

• The I/O problem — reading every object to answer a query is not viable at this scale. The engine must prune aggressively before touching storage.
• The compute problem — the data that does need to be read can still be enormous. The execution layer must scale to match a query's needs for a few seconds, then drop to zero.

A Query Planner addresses the first by using Iceberg metadata to prune terabytes before a single byte is read. A distributed Query Execution system addresses the second by spreading work across Cloudflare's network using Workers and R2 storage.

How the Planner Prunes Data

The most efficient read is the one that never happens. The R2 SQL Query Planner exploits the metadata hierarchy managed by R2 Data Catalog to skip irrelevant data files without inspecting them. The planner walks the Iceberg metadata layers top-down, using summary statistics at each level to build a precise execution plan.

These "stats" exist at two granularities:

• Partition-level stats, stored in the Iceberg manifest list, describe the range of partition values (e.g., earliest and latest day(event_timestamp)) for all files tracked by a given manifest.
• Column-level stats, stored in manifest files, describe each individual Parquet data file. For every column the manifest records min/max values and null counts. A file whose http_status column ranges from 200 to 404 cannot contain a row where http_status = 500, so the planner drops that file. Likewise, a file whose error_code is entirely null can be skipped for a query with WHERE error_code IS NOT NULL.

Walking the Metadata Layers

The planning process proceeds in stages down the Iceberg hierarchy:

1. Table metadata and snapshot — the planner asks the catalog for the current table metadata JSON (schema, partition spec, snapshot log) and selects the latest snapshot.
2. Manifest list pruning — the snapshot points to a single manifest list. The planner reads it and uses partition-level min/max stats to discard whole manifests that cannot satisfy the query's partition filters.
3. Manifest file pruning — for surviving manifests, the planner reads each to enumerate Parquet files and applies column-level stats to eliminate files that cannot match the query's predicates.
4. Row-group pruning — for remaining candidate files, the planner reads Parquet footer statistics to skip entire row groups within those files.

The output is a precise list of work units: individual Parquet files and the row groups within them that are candidates for matching rows. This list is dispatched to the execution system.

Planning and Execution as a Pipeline

For a table with millions of files, waiting for full metadata processing would add unacceptable latency. R2 SQL therefore does not produce a complete plan before starting work. Instead, the planner streams work units to the executor as soon as they are identified.

After two initial fetches (one for the snapshot, one for the manifest list), the planner processes manifests and their data files incrementally, emitting matching work units to an execution queue immediately. Compute nodes begin reading data long before the planner finishes its investigation.

This pipeline is augmented by deliberate ordering. The planner does not stream manifests in arbitrary sequence. Guided by metadata stats and the query's ORDER BY clause, it processes manifests first (using partition stats, e.g., newest timestamp first), and then orders the Parquet files within each manifest using column-level stats. The result is a constantly prioritized stream of work units.

Stopping Early Without Reading Everything

That prioritized stream enables an early-termination strategy. Consider a query like ORDER BY timestamp DESC LIMIT 5. As the execution engine processes work units and returns results, the planner concurrently maintains two stateful pieces of information:

• A bounded heap of the best 5 results seen so far.
• A high-water mark derived from metadata, representing the absolute latest timestamp of any data file not yet processed.

The moment the oldest timestamp in the heap is newer than the high-water mark of the remaining stream, no unprocessed file could possibly produce a result that enters the top 5. The planner halts the pipeline, and a complete, correct result is returned — often after scanning only a fraction of the candidate data.

This mechanism currently requires ordering on a column that is part of the table's partition key. Support for ordering on arbitrary columns is planned for the future.

Distributed Execution

Work units produced by the planner are distributed across Cloudflare's global network for massively parallel processing in Workers, coordinated with R2 for storage. This design gives each query access to the full aggregate bandwidth of the network's edge, and then scales that compute down to zero the moment the query completes or is stopped early.

Execution Model: Row Groups and Workers

R2 SQL breaks query work into units called row groups rather than handling entire Parquet files at once. A single Parquet file typically contains several row groups, and often only a small subset holds relevant data. This granularity lets the engine read just the necessary slices of potentially multi-GB files.

The server handling the user's request becomes the query coordinator. It plans the query, distributes row groups across query workers, and consolidates their results. Because servers across Cloudflare's network undergo maintenance frequently, the coordinator checks Cloudflare's internal API to ensure only healthy servers are assigned work. Connections between coordinator and workers travel over Cloudflare Argo Smart Routing for reliable, low-latency links.

Query workers are the horizontal scaling layer. More workers mean faster processing of large queries spanning many files, as the workload spreads across more machines. Both coordinator and workers run on Cloudflare's distributed network, so compute and I/O resources are plentiful for analytical workloads.

Each worker receives a batch of row groups, the SQL query to run against them, and serialized metadata about the Parquet files. That metadata includes exact byte offsets for each row group, so workers don't have to fetch that information from R2 themselves.

Under the Hood with Apache DataFusion

Workers rely on Apache DataFusion, an open-source Rust analytical engine, to execute queries over row groups. DataFusion’s design centers on partitions: each query splits into concurrent streams, one per partition of data.

DataFusion partitions align naturally with R2 SQL’s row-group model. Each row group can be treated as its own independent partition, allowing fully parallel processing. Since row groups hold at least 1,000 rows, the engine also benefits from vectorized execution—running operations across multiple rows at once reduces query interpretation overhead.

There's a spectrum in query processing: sequential batches favor CPU cache locality and lower interpretation costs, while full parallelism uses many cores for faster completion. DataFusion strikes a balance. Each partition stream processes its rows in efficient batches for cache friendliness, and many of those streams run concurrently to utilize multiple CPUs.

DataFusion also offers strong Parquet support, and Parquet’s columnar layout is built for exactly this kind of engine. Columns are physically separated, enabling selective reads: if a query needs only five of fifty columns, only those are fetched from R2, cutting both read volume and decompression CPU time. DataFusion accomplishes this with ranged reads. Its optimizer further pushes filters down to the file-reading layer, so rows guaranteed not to match are never fully materialized.

Returning Results

Query workers hand results back to the coordinator via the gRPC protocol. Results are represented internally as Apache Arrow arrays, the same in-memory format DataFusion uses during execution. Arrow also provides the Arrow IPC serialization format, intended for inter-process communication rather than storage. Workers serialize results to Arrow IPC and embed them in the gRPC response; the coordinator deserializes them and continues operating on Arrow arrays directly.

What’s Next

R2 SQL currently excels at filter queries, but the roadmap includes several additions over the coming months: distributed and scalable complex aggregations, observability tools to help developers trace query performance, and broader support for Apache Iceberg configuration options.

The team also plans to improve developer experience by letting users query R2 Data Catalogs directly from the Cloudflare Dashboard. Longer term, they’re investigating various index types to speed up queries and enable features like full-text search and geospatial workloads, leveraging Cloudflare’s distributed compute and networking infrastructure.

R2 SQL is now available in open beta. The getting started guide walks through building an end-to-end pipeline that ingests events into an R2 Data Catalog table and queries it with R2 SQL. Feedback and questions are welcome on the Cloudflare Developer Discord.