Pipelines: Streaming ingestion without the Kafka tax

Cloudflare is taking the wraps off Pipelines, a streaming ingestion service that feeds high-volume structured event data straight into R2 object storage. It's launching in open beta today, aimed squarely at developers who want to land real-time data in a queryable format without babysitting a message broker cluster.

The pitch: you don't provision shards, scale metadata services, or manage any infrastructure. If you're on a Workers paid plan, you can point a fetch call or a Worker binding at a pipeline and push through tens of thousands of requests per second. Right out of the box, records get durably persisted, batched into files, and delivered to an R2 bucket you specify. You pay for the bytes processed, not for idle compute or cluster uptime.

Getting a pipeline up is one Wrangler command away:

$ npx wrangler@latest pipelines create my-clickstream-pipeline --r2-bucket my-bucket

...
✅ Successfully created Pipeline my-clickstream-pipeline with ID 0e00c5ff09b34d018152af98d06f5a1xv

And writing a record is equally quick:

$ curl -d '[{"payload": [],"id":"abc-def"}]' 
"https://0e00c5ff09b34d018152af98d06f5a1xvc.pipelines.cloudflarestorage.com/"

At the default settings, a pipeline accepts data from two sources: Workers and an HTTP endpoint. The HTTP path supports adding authentication, configuring CORS headers for browser-based clients, and tuning output file compression and batch cadence. Each pipeline is built to sustain roughly 100,000 records per second on day one, with headroom to grow from there.

Batching without the tiny-file problem

Once records land, Pipelines buffers them durably before writing out files to R2. That buffer matters: dropping millions of tiny files into a bucket is a fast track to slow, expensive queries. The service lets you balance batch size against latency so your query engine isn't munching on a trail of dust when you issue a SELECT. Output files are partitioned by date and time using the standard Hive layout, and file names are ULIDs, so results are time-sorted by construction.

BLOG-2785 Image 1

The files themselves are newline-delimited JSON (NDJSON), which keeps the door open for materializing a stream straight from those files later — Cloudflare hints that R2 may show up as a pipeline source in the future.

The sharding underneath

The architecture splits concerns cleanly. A control plane handles bookkeeping, shard tracking, and pipeline lifecycle events. The data path is built from Durable Objects, each with embedded zero-latency SQLite storage that immediately persists writes before downstream processing kicks off.

The write flow looks like this:

  1. A Pipelines Worker receives records via the fetch handler or a worker binding.
  2. It contacts the Coordinator using the pipeline_id to fetch the execution plan (then caches it to reduce coordinator load).
  3. The plan shards requests across a set of executors that scale read handling.
  4. Those executors re-shard to a second group that owns the writes, starting with persistence into Durable Object storage. The Storage Relay Service (SRS) handles replication for durability and availability.
  5. After SRS, records pass to whatever Transform Workers are configured.
  6. Data is batched, written to output files, and compressed as needed.
  7. Final files are loaded into the target R2 bucket.

Backpressure flows through every stage. The service leans on ReadableStreams — which cross Durable Object boundaries via JSRPC between Durable Objects — and responds with HTTP 429s when buffered bytes exceed a threshold. RPC stubs are reused between Durable Objects to avoid connection churn. Each stage can retry through transient unavailability in Durable Objects or R2.

Deploys don't drop data either. Updating a pipeline spins up a fresh deployment with new shards and Durable Objects, gracefully reroutes requests, and keeps the old deployment writing to R2 until its Durable Object storage drains completely. The old pipeline shuts down only when the last bytes are materialized.

Where Arroyo fits in

Pipelines alone handles ingestion and object storage landing. But raw ingest is rarely the final state — you often want to transform or enrich events on the fly and land them in an open table format like Apache Iceberg. That's the gap Cloudflare is closing by acquiring Stream Processing, a cloud-native engine built in Rust.

Arroyo's origin story tracks the pain of Apache Flink. Flink brought together fault-tolerant, distributed stream processing with stateful dataflow, but its Java API was too low-level for working engineers, and operating a stateful streaming cluster was a full-time job. The Arroyo team set out to build something with Flink's power, but with SQL as the primary interface, and a state backend native to object storage. That simplifies the inherently awkward chore of running stateful pipelines, which behave like bespoke databases you have to keep healthy.

Looking forward, Cloudflare plans to expose Transform Workers interacting with the pipeline as a key handoff point, and to weave Arroyo's SQL windowing and real-time transformations into Pipelines itself — paired with the just-announced R2 Data Catalog, the direction is a distribution of data services pushing compute out to the edge. For now, Pipelines is the first on-ramp for inbound event traffic.

As for cost, nothing extra is charged during this first phase of open beta beyond the standard R2 storage and operation fees. R2 egress stays free, so query engines in any cloud or region can hit the data without racking up transfer charges. Down the road, expect pricing based on bytes ingested and delivered, and Pipelines will eventually land on the Workers Free plan as the beta progresses. Cloudflare says it will give at least 30 days' notice before charging, anticipated no earlier than September 15, 2025.

Roadmap and getting started

The near-term roadmap focuses on deepening the integration between Pipelines and the wider Cloudflare ecosystem. Planned work includes adding Workers as user-defined functions (UDFs) inside Arroyo pipelines, supporting new sources such as Kafka clients, and expanding Pipelines sinks beyond the initial R2 offering.

Another priority is connecting Pipelines with the recently announced R2 Data Catalog. This will let teams write streaming data directly into Iceberg tables and run queries against that data immediately, without relying on external orchestration or separate storage systems.

Resources for developers

For those ready to experiment, the following resources are available:

Alternatively, deploy the example project directly:

$ npm create cloudflare@latest -- pipelines-starter 
--template="cloudflare/pipelines-starter"