Queue Processing: When Your Data Outpaces Your Consumers
Systems that exchange data rarely operate at the same speed. Producer APIs may deliver bursts of information, while consumer services face rate limits, processing overhead, or per-usage costs. Cloudflare Queues, the messaging service built on Workers, now offers two features that give developers greater control over this imbalance: consumer concurrency and explicit message acknowledgment. Both aim to reduce the friction of moving messages between systems without forcing developers to manage the underlying scaling logic themselves.
Autoscaling Consumers to Match Input Volume
When a producer generates messages faster than a consumer can process them, a queue can accumulate a backlog. The new Consumer Concurrency feature addresses this by automatically invoking multiple instances of the configured Worker script to process messages in parallel. This is enabled by default for all queues, with scaling set to automatic.
Cloudflare's autoscaling logic monitors several signals to decide how many consumer instances to spin up: the current number of messages in the queue, the rate of incoming messages, and the ratio of successful to failed consumption attempts. When messages are being processed successfully and the queue remains populated, concurrency increases. If message batches encounter errors, the system scales down.
For scenarios where automatic scaling isn't ideal, developers can set a max_concurrency value via the Cloudflare Dashboard or Wrangler CLI. Capping this value is useful when producer data arrives in bursts, when the downstream API has rate limits, or when higher usage from a data source incurs additional cost. In these cases, limiting concurrency optimizes for cost or API constraints rather than pure processing speed.
A practical example: a fantasy baseball application pulling historical game statistics from a third-party API. A Worker script fetches data and sends each game's statistics to a queue as an individual message. This produces hundreds of thousands of messages. Without concurrency, a single consumer processes each batch sequentially, taking minutes or longer. With concurrency enabled, multiple instances of the consumer Worker process batches simultaneously, while each message contains the player name and a computed quality score stored in a KV store for later access.
Per-Message Control in Your Consumer
The previous Queues behavior treated a message batch as an atomic unit: if any single message in the batch failed processing, the entire batch was resent and reprocessed. For consumers handling messages with varying likelihood of failure, this caused wasted processing cycles on already successful messages, slowing throughput and increasing cost.
Explicit Acknowledgement introduces per-message control. Developers can now instruct the queue whether each individual message was handled successfully or should be retried, regardless of the batch outcome. An acknowledged message will not be resent if the batch fails; only unacknowledged messages are retried. Conversely, a message explicitly marked for retry will be delivered again without affecting the processing of the rest of the batch.
Four new methods are available in the consumer:
.ack()— marks a single message as processed and removes it from the queue.retry()— puts a single message back on the queue for redelivery in another batch.ackAll()— acknowledges the entire batch.retryAll()— retries the entire batch
In practice, this helps when a portion of incoming data is malformed. In the baseball scenario, if an upstream API occasionally sends data that fails parsing, the consumer can call .ack() for the successfully parsed messages and .retry() only for the bad ones, instead of reprocessing the whole batch.
More Messages Per Second
Alongside these new controls, Cloudflare engineers have increased the base throughput of each queue. The sustained rate has quadrupled from 100 to over 400 messages per second in recent months. The team continues to target further throughput improvements to accommodate growing workloads.
What's on the Roadmap
Currently, a queue consumer must be a Workers script. Cloudflare plans to broaden the consumer options in the near term.
R2 as a Direct Consumer
Coming soon, queues will support writing directly to an R2 bucket, selectable from the Dashboard without requiring a Worker script in between. This eliminates the code and maintenance overhead of a dedicated consumer script for cases where the destination is simply an R2 bucket.
HTTP Pull API
For existing infrastructure outside of Cloudflare, Queues will offer an HTTP API endpoint for each queue. Any consumer can request batches of messages, process the data, and send an acknowledgment to signal the queue to move on to the next batch.
Cloudflare Queues is currently in Open Beta. The team's focus remains on reducing message processing time, minimizing the code developers must manage, and providing finer-grained controls over the processing pipeline. Developers can get started through the official documentation.
