From message queues to durable orchestration

Cloudflare operates as a reverse proxy for roughly a fifth of the web, and its Workers platform, R2 storage, and CDN services put it in a unique position on the Internet's request path. Since 2010, the company has worked with the National Center for Missing and Exploited Children (NCMEC) to support the identification and removal of child sexual abuse material (CSAM). Public reports, customer submissions, and alerts from trusted organizations flow through Cloudflare's Trust & Safety team, which triages CSAM-related cases with top priority and forwards them—along with relevant files and supplemental information—to NCMEC.

Generating and submitting those reports involves multiple dependent steps and error handling paths. Under the original queue-based architecture, that complexity became a maintenance burden. Cloudflare replaced it with a Workflows-based orchestrator that provides structured execution, built-in retries, and full observability.

Why the old queue-based design fell short

The NCMEC reporting system was designed in early 2024, before Cloudflare Workflows existed. It relied on Worker Queues to manage asynchronous tasks and structured the pipeline around them. The goal was reliability, fault tolerance, and automatic retries—but without an orchestrator, state management, retry semantics, and inter-queue messaging all had to be handled manually.

Each report moved through five discrete steps, with a separate queue handling each one:

  1. Validate input: Ensure the report contains all required details.
  2. Initiate report: Call the NCMEC API to create a report.
  3. Fetch impounded files (if applicable): Retrieve files from R2.
  4. Upload files: Send files to NCMEC via API.
  5. Finalize report: Mark the report as complete.

When a step failed, the system revolved the message several times before marking the report as failed. But not all errors were transient—bad input or an unexpected response shape from an external API wouldn't resolve with retries, leaving reports stuck in intermediate states. Debugging those required manually combing through logs across multiple queues.

Recovery was handled by a "Reaper" cron job that ran hourly and resubmitted failed reports. Since a report could fail at any step, the Reaper had to infer which queue failed and send a message to restart processing from that point. The resulting pain points were familiar to anyone who has operated a distributed pipeline without clear orchestration:

  • Debugging was laborious: Tracing one report meant correlating logs from several queues.
  • Retry logic was inconsistent: Some queues had their own retry logic; others depended on the Reaper, leading to unpredictable behavior.
  • State was opaque: There was no straightforward way to know whether a report was mid-pipeline or lost, short of reading through logs.
  • Operational overhead was high: Engineers frequently had to manually inspect and resubmit failed reports.

The queues served well as a message bus, but they weren't designed to orchestrate a multi-step workflow. The team had effectively built cooperating services on top of messaging and relied on convention and glue code to hold them together. It worked—mostly—but it was hard to reason about and easy to break.

Workflows: a single orchestrator for the whole pipeline

Cloudflare Workflows gave the team an alternative: instead of wiring queues together, a single workflow definition controls the entire process from start to finish. If a step fails, the workflow resumes from the point of failure without re-running earlier work—no re-parsing files, no duplicated uploads.

The rearchitected system follows a clear sequence:

  1. Create the report: Validate the incoming report and initiate it with NCMEC.
  2. Check for impounded files: If files are associated with the report, proceed to file collection.
  3. Gather files: Retrieve impounded files from R2 and prepare them for upload.
  4. Upload files to NCMEC: Submit each file through the NCMEC API.
  5. Add file metadata: Attach hashes, timestamps, and other metadata to the report.
  6. Finalize the report: Mark the report as complete once all files are processed.

The dashboard provides live visibility into every workflow: active runs, completed runs, per-step status, and failure points. Steps can be retried or workflows terminated directly from the UI. Failed steps are retried with exponential backoff, which smooths over transient issues like flaky APIs without requiring manual intervention.

The shift from queue orchestration to Workflows changed how the team handles NCMEC reports. What was a loosely connected set of queue consumers is now an explicit, retryable, observable process with straightforward debugging and monitoring.

Building your own workflows

Developers who have an existing Workers application that has grown into a multi-step process can typically wrap their code in a Workflow with minimal changes. Workflows can read from R2, write to KV, query D1, and call external APIs just like any other Worker—but they are built to coordinate asynchronous, long-running tasks. The Workflows developer documentation and starter project are the quickest way to begin.