Building Cloudflare on Cloudflare

Why Cloudflare is Rebuilding Its Own Stack on Its Developer Platform

Cloudflare’s web-facing products—security, performance, and application services—currently process around 46 million HTTP requests per second. For over a decade, these systems were built as native Linux services, but the company has been steadily shifting components onto its own Workers developer platform. The motivation is straightforward: faster iteration, stronger isolation, and a better experience for the engineers who maintain these systems.

The core architecture is a chain of HTTP-speaking proxies, each with a distinct job. Historically these were built on NGINX and Lua, but many have since been rewritten in Rust, most notably Pingora. One key component is the "FL" service, which performs front-line processing—applying customer configuration to determine how each request should be handled and routed.

This arrangement has served Cloudflare for more than a decade. Independent services can be developed, deployed, and scaled separately, and traffic can be routed dynamically based on load or cache efficiency. But there’s a rising cost: adding new services to the chain incurs latency at every HTTP boundary, and the team has reached the practical limit of how many hops can be added without sacrificing performance.

Rather than proliferating services, most product logic has been consolidated into FL. That keeps latency down, but it creates a different problem—the service has grown so complex that only a small, highly experienced team can safely operate it.

The developer experience has suffered as a result. Reproducing the environment locally requires custom tooling and tuned Linux kernels. Code changes that would seem simple can hit platform limitations—for instance, it’s impossible to perform I/O in many portions of the HTTP response-processing path, forcing engineers to preload resources speculatively. Deployments are slow and batched, so isolating the impact of any single change is difficult, and while the code is modular, production runtime lacks real isolation or sandboxing.

Bringing Workers In-House

Cloudflare began experimenting with Workers for internal use shortly after the platform launched in 2017. By 2023, a substantial number of products—including Zero Trust, R2, KV, Turnstile, Queues, and the Exposed Credentials Check—already run at scale on Workers, handling every request those products process. The push now is to give internal engineers the same developer experience that external customers already enjoy on the platform.

It’s important to distinguish between two kinds of Workers at Cloudflare. The first is the customer-facing kind: code deployed by external users to run on routes they control. Internal engineering teams also use these, treating themselves exactly like any other customer. The second kind is Cloudflare-only: an internal Worker not tied to a single customer, but triggered for all customers who need a particular piece of logic. The remainder of this discussion focuses on those internal Workers, which run in response to requests arriving from many Cloudflare customers.

A First Generation with Real Limits

Cloudflare first integrated internal Workers into the request path in 2019 via an ordered chain that executes before any customer scripts run. The expected latency concern didn’t materialize because these Workers don’t sit at a service boundary—they execute inside the same runtime as adjacent steps, with control shifts happening via memory references rather than HTTP marshaling. Each additional step still adds overhead, but far less than a full service hop would.

That initial integration delivered immediate benefits. Workers’ sandbox model removed the possibility of cross-request or cross-customer side effects, and teams gained the ability to deploy independently without coordinating schedules.

Yet the implementation had meaningful gaps. Internal Workers could only execute at one point in the request lifecycle, so they couldn’t influence services running earlier—such as the WAF. Publishing them required a special internal API and credentials, which closed the door on the rapidly improving public tooling, including Wrangler. Observability was sparse, with limited metrics and logs that made debugging painful. Even so, a dozen products—including Zaraz, the Cloudflare challenges system, Waiting Room, and performance features like Image Resizing, Images, Mirage, and Rocket Loader—adopted internal Workers to handle parts of their logic.

Testing the Limits with Flame

Late in 2021, Cloudflare began asking how far the platform could be pushed. Specifically: could the logic currently locked inside the NGINX-based FL service be migrated to the developer platform, and if not, where exactly were the barriers? To answer, the team built a prototype called Flame, which routed traffic directly from the TLS ingress service to the Workers runtime, bypassing FL entirely.

Flame worked, but only barely. The prototype exposed several fundamental capabilities that were missing. There was no way for Workers to reach Cloudflare’s internal DNS infrastructure or the customer configuration database, nor could they emit request logs into the data pipeline used for analytics and billing. Stateful caching between requests wasn’t possible, and Workers couldn’t send requests directly to customer origins or the cache without going through the existing full proxy chain.

Equally important were the developer experience failures. Because internal Workers required a private API for deployment, engineers couldn’t use the same Wrangler-based workflows that customers rely on. Metrics and traces weren’t feeding Cloudflare’s standard observability systems, leaving the team effectively blind in production. And there was no mechanism for staged, gradual rollouts of updated code.

Flame was a successful experiment in that it clearly identified the gaps. The question then became whether those gaps could be closed—and what it would take to make the developer platform the default home for Cloudflare’s own infrastructure.

Building internal services on the developer platform

Our initial prototype revealed several gaps between what internal services needed and what the developer platform could provide. Rather than maintaining separate tooling, we chose to extend the platform's existing capabilities so that both internal and external developers benefit from the same improvements.

Deploying internal Workers with standard tooling

Internal services previously relied on special APIs for deploying Workers, justified for "security reasons." After review with our security team, we determined that our existing API already had strong protections for identifying who published a Worker. The missing piece was a secure registry of accounts authorized to use privileged resources. We initially hard-coded these permissions into the API service, later replacing this with a more flexible permissions control plane.

A crucial distinction exists between publishing and deploying a Worker. Publishing pushes the Worker to our configuration store, creating a new artifact for each version. The Workers runtime uses a capability-based security model: each published script is bundled with bindings that represent the resources it may access. This ensures that only accounts with appropriate permissions can grant capabilities to scripts.

Deploying, by contrast, hooks the Worker up to be triggered by incoming requests. For customers, this means attaching to a route. For internal services, this means updating a global configuration store with the specific artifact identifier to run.

Once we could use wrangler to build and publish internal services, a new problem emerged: we needed to know the artifact identifier to deploy. A simple update to wrangler solved this by outputting the necessary debug information. This change unlocked the full development workflow — engineers could check out code and immediately use wrangler test and wrangler dev in a realistic environment.

Logging, metrics, and observability

Our data pipeline starts as Cap'n Proto messages from the network, feeding services for customer analytics, operations, DoS protection, and billing. To push log data from inside a Worker to this pipeline, we added a new binding to the logfwdr service. This work later became the foundation for the Workers Analytics Engine bindings, giving customers the same structured logging capabilities.

Three observability pillars matter most at Cloudflare:

• Unstructured logs ("syslogs") ingested into systems like Kibana for searching and visualization.
• Metrics emitted as numbers (CPU usage, requests handled) and ingested into our massive Prometheus system for alerting and trend analysis.
• Traces using Open Telemetry-based systems to record detailed component interactions and timing.

Our observability team built initial support for all three for internal Workers, providing endpoints that Workers could push to. We wrapped this in a library called flame-common to abstract away the mechanics:

import { ObservabilityContext } from "flame-common";
 
export default {
    async fetch(
        request: Request,
        env: Env,
        ctx: ExecutionContext
    ): Promise<Response> {
 
        const obs = new ObservabilityContext(request, env, ctx);

        // Logging to syslog and kibana
        obs.logInfo("some information")
        obs.logError("an error occurred")
 
        // Metrics to Prometheus
        obs.counter("rps", "how many requests per second my service is doing")?.inc();
 
        // Tracing
        obs.startSpan("my code");
        obs.addAttribute("key", 42);
    },
};

This API required passing an ObservabilityContext around to emit events — an awkward design. Resolving this drove our recent addition of AsyncLocalStorage support to the Workers runtime. While the current system works, the internal implementation isn't as efficient as we'd like. We're now adding native support for emitting events, metrics, and traces directly from the runtime, following the same approach as the Workers Analytics Engine: hook into internal systems but also expose the capability to customers.

Accessing internal resources over Cap'n Proto RPC

Moving more logic onto the developer platform required internal Workers to reach services like Quicksilver (our configuration store), DNS, and DoS protection systems. Since many of these systems use Cap'n Proto for serialization and communication — and are implemented in Go or Rust with solid client support — we added Cap'n Proto RPC support to the Workers runtime.

Each service listens for connections from the runtime and publishes a schema for communication. The runtime converts data between JavaScript and Cap'n Proto according to a schema bundled with the Worker at publication time. This makes calls to internal services simple. Here, our DNS service identifies the account owning a hostname:

let ownershipInterface = env.RRDNS.getCapability();

let query = {
  request: {
    queryName: url.hostname,
    connectViaAddr: control_header.connect_via_addr,
  },
};

let response = await ownershipInterface.lookupOwnership(query);

Caching with volatile in-memory storage

Workers provide no guarantees of state between requests, which complicates caching. We added a new internal binding providing a "volatile in-memory cache" that is shared between Workers wherever efficient. The semantics are:

Using the cache wraps a block of code:

const owner = await env.OWNERSHIPCACHE.read<OwnershipData>(
  key,
  async (key) => {
    let ownershipInterface = env.RRDNS.getCapability();

    let query = {
      request: {
        queryName: url.hostname,
        connectViaAddr: control_header.connect_via_addr,
      },
    };

    let response = await ownershipInterface.lookupOwnership(query);
    const value = response.response;
    const expiration = new Date(Date.now() + 30_000);
    return { value, expiration };
  }
);

This drastically reduces external resource calls. Future improvements include background refreshes to reduce P99 latency and better observability into usage and hit rates.

Controlling egress

Previously, the Workers runtime always communicated with the FL service, which handled some product logic after Worker execution. In many cases this added unnecessary overhead. We've now given internal Workers control over how requests egress — either directly to our cache systems or directly to the Internet, bypassing FL when no logic is needed there.

Deployment with health checks

Many teams using internal Workers had built their own deployment systems, but these relied on manual steps: copying identifiers and triggering advancement at the right time. We built a new deployment system on several principles:

• Git-controlled deployments: merges to a staging branch deploy to an internal-only environment; merges to a production branch deploy to production.
• Progressive deployment: releases advance from low-impact pre-production systems through stages of increasing impact.
• Health-mediated advancement: between each stage, end-to-end tests run, metrics are reviewed, and a minimum time elapses. Failures pause or revert the deployment automatically, without human intervention.

This frees developers to focus on system behavior rather than deployment mechanics. These systems now run in production for many internal Workers, and further improvements are planned.

From prototype to production

The prototype demonstrated which capabilities were missing from the developer platform. We've added those capabilities and now run relatively small internal components on them in production. If we were building our application security and performance products from scratch today, we could build them on the platform.

Having a capable platform, however, is different from migrating existing systems onto it. We're in the early stages of migration: real traffic runs on the new platform, and we expect to move more logic — and some full production sites — off the FL service within the next few months.

We're also determining the right module structure. The platform lets us split logic across many Workers that communicate efficiently. Finding the correct subdivision — matching development processes while keeping code maintainable and throughput high — remains an open question.

Migration in practice, not in theory

History suggests we shouldn't underestimate the difficulty of a rewrite. Large legacy systems hide years of accumulated assumptions, and a fresh implementation risks recreating problems the old architecture solved long ago. A rewrite or migration only makes sense if it offers a clear win in developer experience, reliability, or performance — and if it can proceed without stalling feature development for even a moment.

This isn't speculative. Cloudflare has rebuilt critical infrastructure before. Quicksilver, the configuration distribution system, has been fundamentally reworked multiple times, moving first from Kyoto Tycoon and later migrating its datastore from LMDB to RocksDB. The HTML rewriting code was also rebuilt to leverage the safety and performance of newer technologies.

More tellingly, this exact system has already survived a full architectural rewrite. The original performance and security proxy was written in PHP, and that implementation was retired in 2013. That effort succeeded because it ran without downtime: the new system was so much easier to build that its developers could keep pace with changes to the old one. Once it was mostly complete, it handled requests live, falling back to the legacy system when it couldn't process one. Only when enough logic had been ported could the old system be switched off.

Author: Dane Knecht
Date:   Thu Sep 19 19:31:15 2013 -0700

    remove PHP.

A harder problem this time

Our systems are far more complex than they were in 2013, so a similar big-bang approach isn't on the table. The strategy is incremental: identify separable parts of the system that offer concrete near-term benefits, migrate those to new architectures, and feed the lessons back into platform and tooling improvements before picking the next target.

Modularity is central to this plan. The system will be modified by many teams, so strong boundaries between code modules are essential. Engineers should be able to reason about the system locally, without needing global knowledge of the entire codebase.

One advantage of the developer platform is that we don't need to publish a single version of our software. We can run several systems concurrently, each tuned for different use cases, without forcing users onto one implementation.

The internet rarely matches its specification. Standards and RFCs describe intended behavior, but real-world traffic frequently turns up undocumented edge cases. Whenever a migration changes that behavior — even unintentionally — we risk breaking assumptions someone has relied on. That doesn't rule out such changes, but it does mean we have to be deliberate about them, understanding the impact in advance to minimize disruption.

Our testing infrastructure is therefore critical. We already run extensive tests on both the software and the network, but we're building capabilities to test every edge case in production, before and after each change. This lets us experiment with more confidence and decide, for each migrated piece, whether compatibility needs to be bug-for-bug — and if not, whether anyone deserves a warning. This approach has precedent: when we rebuilt the DNS pipeline to run three times faster, we built similar tooling to verify that the new system behaved identically to the old one.

Some lessons will surprise us, no doubt. Those findings will improve the developer platform's capabilities and ease of use, and the scale of our systems will expose bottlenecks that might otherwise stay hidden. We'll share progress and unexpected findings in future posts.

Getting involved

If you'd like to hear more about this work, or tell us what capabilities you want from the developer platform, you can reach us on Discord.