Announcing the Cloudflare Data Platform

Earlier this year, Cloudflare launched the public beta of R2 Data Catalog—a fully managed Apache Iceberg catalog designed to sit atop R2 object storage. Today, that foundation expands into a complete data platform with three new offerings: Cloudflare Pipelines for event ingestion and transformation, ongoing table maintenance in R2 Data Catalog, and R2 SQL, a distributed query engine built for petabyte-scale analytics.

The common thread across all three products is Cloudflare's global infrastructure. Each runs directly on the company's edge network—no fixed clusters, no centralized infrastructure to manage. The platform is also built around open standards, so it works with any Iceberg-compatible engine—PyIceberg, DuckDB, or Spark—and can connect with external platforms like Databricks and Snowflake. And because R2 notoriously charges no egress fees, users can move data out without penalty once it's stored.

For a step-by-step walkthrough of building a data pipeline from end to end, the Data Platform tutorial covers creating a Pipeline, ingesting events into an R2 Data Catalog table, and querying it with R2 SQL.

Why a data platform?

R2 launched in 2021 with a then-radical pricing model: zero egress fees, eliminating the bandwidth costs that made moving data between clouds so expensive. Over time, Cloudflare noticed an increasing number of users storing analytical workloads in R2, enabled by the growing industry adoption of Apache Iceberg.

Iceberg is a table format—a metadata layer over object storage that provides database-like capabilities such as ACID transactions, schema evolution, and update support. With wider Iceberg adoption, users were no longer locked to a specific query engine. But they still faced the problem of managing their own metadata infrastructure and optimizing tables over time. That gap led to R2 Data Catalog, which handles Iceberg metadata as a managed service.

Data storage and metadata management alone didn't solve everything. Three practical gaps remained: getting data into tables efficiently, keeping those tables query-optimized as they grow, and running queries without standing up an additional engine or paying another cloud. The three components of the Data Platform address these challenges directly.

Cloudflare Pipelines: getting data in

Analytical tables are made of streams of events from sources like server logs, mobile apps, or IoT devices. Before Iceberg can make sense of them, those events must be ingested, structured to a schema, and written into object storage. Pipelines, built on the Arroyo stream processing engine that Cloudflare acquired earlier this year, handles that entire path via SQL transformations followed by writes to R2 or R2 Data Catalog.

Pipelines is organized around three core objects:

  • Streams are the entry point for data—durable, buffered queues that accept events via an HTTP endpoint or directly from a Worker binding.
  • Sinks define where data lands, either in an R2 Data Catalog table or as raw files (JSON or Apache Parquet) on R2. Sinks can be tuned for low-latency ingestion with frequent small writes or for query performance with larger, less frequent files. Either way, ingestion is exactly-once: no duplication, no drops.
  • Pipelines bind streams to sinks with SQL transformations, giving you the option to validate, schematize, or otherwise reshape events before they ever touch storage.

That "shift left" capability means users can push processing into the ingestion layer. Practical uses include extracting structured fields from arbitrary JSON with JSON functions, filtering out unwanted events or splitting them into schema-specific tables, redacting sensitive values with regexes, or unrolling nested arrays and objects into individual events.

The initial Pipelines release supports stateless transformations. Later updates will tap into Arroyo's stateful processing to unlock aggregations, incrementally updated materialized views, and joins. In the meantime, the service is available as an open beta through the Cloudflare dashboard, Wrangler, or the REST API.

Pipelines itself is not billed during the beta, though R2 storage and operation costs from sink writes apply at standard rates. Future pricing is projected to factor in data read, data transformed via SQL, and data delivered.

R2 Data Catalog: making tables faster

The R2 Data Catalog open beta has already attracted attention since April—DuckDB added native support and integrations like marimo notebooks emerged. Creating a catalog is as simple as issuing a couple of Wrangler commands; there's no database cluster to stand up, no object storage to connect, no infrastructure to administer. The result is a petabyte-scale data lake queryable by any Iceberg engine.

But a catalog that only stores data leaves a maintenance problem unresolved. As ingestion continues, the number of underlying data files grows—especially with low-latency sinks that write small files frequently. Each file in a query must be listed, read, and downloaded, and metadata overhead from too many small files can quickly dwarf query time.

The answer is compaction: automatically rewriting small files into larger ones to reduce metadata overhead and speed up queries. Compaction is now available in R2 Data Catalog and can be enabled per catalog. The initial release focuses on small-file compaction, with additional strategies planned. R2 Data Catalog remains free during beta; future pricing is still being determined.

R2 SQL: querying in place

Historically, extracting value from stored data has meant setting up and managing a query engine—DuckDB, Spark, or Trino—adding operational overhead to every analytical workload. With R2 SQL, released today as an open beta, queries run directly on Cloudflare's infrastructure through Wrangler. There's no cluster to manage, and the engine is built from the ground up for tight coupling with R2 Data Catalog metadata and R2 storage.

That integration is not cosmetic. The query planner uses the rich statistics resident in R2 Data Catalog metadata rather than simple storage scanning, which opens the door for more sophisticated optimizations such as auxiliary indexes or more advanced analytical functions down the line. Because Cloudflare schedules compute globally, data is processed where it lives, eliminating the pull-to-central cluster pattern that usually defines analytical workloads.

The service is completely serverless from the user's perspective. The initial beta focuses on filter queries, with more SQL features like complex aggregations to follow. R2 SQL usage is not billed during the beta, but standard R2 storage and operation charges apply. Eventually, pricing will be based on the volume of data scanned by queries.

What's next

The first half of 2026 brings a slate of additions across the Data Platform:

  • Integration with Logpush so Cloudflare logs can be transformed, stored, and queried natively
  • User-defined functions authored in Workers, plus stateful processing for streaming transformations
  • R2 SQL expansion to support aggregations and joins

A hands-on way to test the platform today is the step-by-step tutorial, which walks through building a full analytics system with Pipelines, R2 Data Catalog, and R2 SQL. Feedback is welcome on the Developer Discord.