Rebuilding a migration pipeline on Workers, Queues and Durable Objects
Super Slurper, Cloudflare's tool for transferring data between object storage providers and R2, has moved petabytes of data since launching. The original implementation, however, scaled poorly. It inherited its architecture from SourcingKit, a Kubernetes-based image import tool that ran in a single core data center. As Super Slurper grew, tens of pods sharing compute and bandwidth with Cloudflare's control plane became the limiting factor. The task itself is simple — list source objects, queue them, copy them — but the resource contention made scaling painful.
The team needed more concurrency, not more CPU or memory. Rather than continuing to add Kubernetes replicas, they rebuilt the entire service on Cloudflare's Developer Platform: Workers for lightweight compute, Queues for asynchronous task distribution, Durable Objects for state, and Hyperdrive for access to legacy data. The proof of concept took about a week to build and, importantly, validated the approach: it ran slower than the old system for a few hundred objects, but matched and exceeded it once migrations reached millions of objects.
From Postgres queue to fully distributed processing
The original pipeline relied on listing objects from the source bucket, pushing them to a Postgres-backed queue (pg_queue), and steadily pulling from that queue to copy objects. A Worker was introduced to move the actual data copying out of the core data center, but the orchestration remained bound to Kubernetes.
The new design keeps the same conceptual flow — list, queue, transfer — but pushes each stage into components that scale independently:
- Cloudflare Queues handle asynchronous transfer messages and auto-scale with the number of objects.
- Workers run the lightweight compute tasks, optimizing for geographic locality and latency.
- SQLite-backed Durable Objects replace the single Postgres instance as a fully distributed database.
- Hyperdrive provides a fast path to the original Postgres database, kept as an archive for historical job data.
Processing layer: tracking jobs through the pipeline
Migration requests reach the API Worker, which records details in the database and enqueues a message on the List Queue. The List Queue Consumer then pulls object listings from the source bucket, applies filters, stores metadata, and immediately enqueues transfer tasks onto the Transfer Queue. This immediate batching maximizes concurrency, but a built-in throttle prevents message flooding if a dependent system fails.
Transfer Queue Consumers pick up object transfer messages and lock each object key in the database before copying, guaranteeing single processing even under retries. Larger objects are split into multipart uploads. Transient failures retry automatically; persistent ones route to a Dead Letter Queue for manual review.
A Lifecycle Queue Consumer monitors all in-flight transfers, watches for stragglers, and marks the job complete once every object has moved.
Database layer: Durable Objects and the legacy bridge
Each account gets a dedicated Durable Object to track migration state — bucket names, user options, job status. Large migrations add a Batch DO that records transfer state and object metadata for everything queued for copy. At the scale of billions of objects, storage fills quickly; a sharding strategy distributes load and works around the SQLite DO's 10 GB limit, and per-object records are deleted as transfers complete. Hyperdrive steps in to keep two years of pre-existing migration history accessible from the original PostgreSQL store.
The four technology choices break down as:
The payoff: five times faster transfers
The team benchmarked a migration of 75,000 objects from AWS S3 to R2. The old implementation completed in 15 minutes and 30 seconds; the rebuilt service finished the same job in 3 minutes and 25 seconds.
Once production traffic moved to the new architecture in February, real-world gains appeared and varied with object size distribution. The performance jump has been substantial — 35% of all objects Super Slurper has ever copied moved in just the last two months, suggesting users are transferring more data more quickly than historically.
Handling the hard part: duplicate messages
The new architecture introduced duplicate message challenges inherent to distributed queues. Queues provide at-least-once delivery, so consumers can see the same message multiple times. Failures also manufacture duplicates: if a Durable Object request fails after an object has already moved, the retry may reprocess the same object.
The team adopted four complementary safeguards to keep transfers idempotent:
- Sequential listing is assigned sequence IDs so duplicate listing operations are detectable and unnecessary retries can be short-circuited. This matters because listing batches are enqueued without waiting for database and queue operations to finish.
- Object keys are locked when a transfer starts, preventing parallel transfers. Once the copy succeeds, the key is deleted; if the message reappears later, the absence of the key proves the object is already done.
- Database transactions keep counters consistent. Failed unlock or insert operations leave counts untouched so the retry still works.
- A final check verifies that the object already exists in the destination and was written after the migration's start time — if so, it is assumed transferred and safely skipped.
Scaling migrations with Workers and Durable Objects
The migration process itself is orchestrated by a set of Workers services. A dispatcher Worker fans out work to Durable Object instances, each responsible for a slice of the migration. The queue of batch operations is backed by Cloudflare Queues, so the orchestration layer stays resilient and out of the way of the hot path: moving bytes.
This architecture is resilient by design. Individual batch operations have retries with backoff. When they finally exhaust their retries, they get sent to a dead-letter queue for inspection. That means a handful of failures does not stall the entire migration, and at the end of a run an operator can exactly see which slices failed and why.
API surface shrinks while coverage grows
The latest revisions also simplified the API. Instead of constructing requests around operation IDs and mapping statuses, you can issue a migration against a source bucket, and parse the response — which includes the full configuration for each object — to identify the source account owner, provider, and so on. That means you can audit, programmatically discover, and automate end-to-end bucket migrations, rather than clicking through a dashboard.
We have also extended support to any S3 compatible provider. This follows the same pattern, so Super Slurper becomes the control plane and the network backbone for multi-cloud data replication.
Future directions
Concurrency is currently capped to three simultaneous active migrations per account. Removing that ceiling is on the roadmap. The immediate benefit should be obvious: splitting object prefixes into separate parallel migrations can multiply per-bucket transfer rates dramatically, without any extra machinery on the customer side. Doing so is trivial when migration definitions are fully represented in the API.
We are only at the start of making Super Slurper a programmatic, automated migration layer.
