Why Cloudflare Simulates Its Own Edge
Cloudflare's edge network handles roughly 20 million requests per second across more than 25 million Internet properties. That scale is made possible by a homogeneous architecture: nearly every server in the network runs every product Cloudflare offers. But homogeneity cuts both ways — each new product or code change touches the entire fleet, and the operational blast radius of any mistake grows accordingly.
Originally, Cloudflare operated only at the application layer with CDN and DDoS protection, then added transport-layer offerings such as Spectrum and Argo, and has since expanded to the IP layer with Magic Transit. All of these run on every edge machine. With that much shared infrastructure, the company needed a way to test changes under realistic conditions without risking customer impact. Its answer is SOAR — Simulation for Observability, reliAbility, and secuRity — a dedicated data center that runs the same software stack as production but carries no production traffic.
SOAR is organized into three server roles: product servers running the software under test, plus simulated end-user and origin servers that generate synthetic customer behavior. All traffic stays within a tightly controlled LAN environment rather than traversing the public Internet, which gives engineers flexibility to test network features like bring-your-own-IP (BYOIP) products. The simplest simulations run network benchmarks through the product servers to evaluate how well a given product performs.
Consider Magic Transit, which provides IP-layer DDoS protection and traffic acceleration. Customers bring their own IP ranges, Cloudflare advertises them, and clean traffic is GRE-encapsulated back to the customer's endpoints. Simulating this product requires more than just sending packets — it means evaluating data plane configuration, BGP readiness, ECMP route programming, health probe correctness, and whether the DDoS system drops malicious traffic as designed. It also means catching corner cases that only appear at scale, and detecting whether unrelated team changes break the product.
To simulate Magic Transit, SOAR creates a virtual customer. If Alice onboarded the range 192.0.2.0/24, SOAR runs an iperf server on an origin server to represent her service, configures a product server to filter traffic toward that prefix, and sets up GRE encapsulation to a specified endpoint. Routing rules direct test traffic through the product server, and test clients act as eyeballs to measure correctness, performance, and resource usage.
From Static Tests to Coordinated Simulation
Before SOAR, Cloudflare had a performance simulation system built on SaltStack, its configuration management tool. The system ran statically defined test cases against Cloudflare-owned HTTP sites around the clock, producing Prometheus metrics such as requests per second and latency for daily monitoring. But as the number of products and simulation cases grew, the approach hit its limits.
The fundamental problem was resource isolation. Cloudflare's products are not virtualized — the company maximizes edge performance by running them natively. That means simulations must clean up after themselves; otherwise, leftover routes and configurations from one test can conflict with another. Engineers once encountered a scenario where different simulations required different source IP addresses to the same destination, forcing teams to modify their test applications or coordinate with each other's code — an unacceptable burden.
Static configuration also could not handle complex test scenarios. Magic Transit customers often maintain multiple GRE tunnels with ECMP hashing across paths, plus failover logic between data centers. Validating that failover works correctly requires injecting errors only after test traffic is already flowing — something impossible to express in a static SaltStack definition.
There was also an engineer-friendliness problem. Adding a simulation meant submitting a Salt PR to the SRE-managed repository, waiting for the static loop to reach it (potentially hours), and having no way to trigger tests per branch build or retrieve timely performance feedback. The system was unmanageable at scale.
SOAR Architecture
SOAR redesigns this as an internal coordinator service with two main responsibilities. First, it provides an interface for engineers to submit one-time simulations from laptops or CI pipelines and to view past results. Second, it dispatches tasks to simulation servers and ensures isolation — the simplest policy being that no two simulations ever run on the same server concurrently.
The coordinator supports two simulation types:
- One-time simulations: run ad-hoc, primarily per pull request, to ease development testing and integrate with the CI system.
- Repetitive simulations: stored persistently and executed periodically for daily monitoring.
The coordinator is protected by Cloudflare Access, restricting access to employees only. Each simulation server runs an agent that executes two kinds of tasks:
- System tasks: change system-wide configurations — routes, links, addresses, ipsets, iptables — and are reverted after each simulation terminates.
- User tasks: run the benchmarks under evaluation, isolated in cgroups, terminated if they exceed an execution budget, and monitored by Cadvisor with metrics flowing to Prometheus and Alert Manager.
SOAR itself runs in a dedicated environment with production-grade hardening: no engineer access except through approved tools, standard security enforcement, and standard alerting. That operational discipline makes SOAR a reliable source of anomaly detection rather than an experimental sandbox.
Customer-Specific Simulations
One of SOAR's significant capabilities is simulating a particular customer's configuration. Since every edge server already knows how to dispatch incoming customer packets, simulating a specific customer reduces to mocking routing and domain translation on the simulated eyeballs and origins so they direct traffic to the correct product servers. Because all servers sit in the same broadcast domain, a direct routing rule and a static /etc/hosts entry suffice.
This design cleanly separates customer configuration management from the simulation service: the product teams own their configurations, and any change automatically reflects in simulations without additional coordination. The payoff is twofold — customers get assurance that their configurations are battle-tested before going live, and during escalations, engineers can bisect problems with unrelated factors already ruled out.
From Release Pipeline to Production Network
SOAR's components are written from scratch in Golang on Linux servers. The prototype and first engineering use case took about three engineer-months to build. Building in-house, rather than adopting mature external platforms for scheduling, isolation, and monitoring, makes it easier to absorb new requirements from engineering teams quickly.
The simulation service is now being wired into Cloudflare's release pipeline. A new product version first goes to dogfooding data centers for internal trial, then to canary data centers with limited customer traffic. If all goes well, it propagates within about an hour through three tiers of larger data centers, with tier-3 receiving changes an hour before tier-2, and tier-2 before tier-1. SOAR adds a simulation gate before dogfooding: changes are deployed to the simulation environment first, where engineering teams select which simulations to run. Dogfooding only begins if there is no functional breakage or performance regression, the latter judged by Prometheus metrics where each team defines its own query. Periodically run simulations also catch issues not tied to a specific product, such as a Linux kernel upgrade, with SREs notified asynchronously.
Real-World Simulation Cases
Validating Data-Center-Specific Releases
A Magic Transit release adding physical network interconnect (PNI) support exposed a weakness in the standard rollout. With PNI, customer data centers physically peer with Cloudflare routers, but those PNI sites are typically not the same as dogfooding or canary centers. Worse, a PNI data center becomes a choke point for a customer's traffic, with no other center able to take over its role if it fails.
Figure 6. Magic Transit traffic flow without PNI.
Figure 7. The PNI data center becomes a single choke point for customer traffic.
SOAR lets engineering configure a regular server with PNI information so it behaves as if it runs in a PNI data center. Simulated eyeball and origin traffic runs against that server to check for functional breakage before any production rollout.
Figure 8. A server configured with PNI information acts as the target for simulated traffic.
This approach caught real problems early, including a checksum offloading defect that produced TCP packets with wrong inner checksums, causing drops at the origin. The issue only surfaced on production hardware, not in virtualized integration tests. The simulation served as a success indicator while fixes were tried until packet flow returned to normal.
Continuous Edge Performance Monitoring
All simulations run on the same shared stack as other teams' products, so an unhealthy simulation result may not relate directly to the product under test. Even so, continuous runs increase the odds of spotting issues early or providing a fast hint when an incident emerges. One performance dashboard showed HTTP request throughput dropping by 20 percent; the cause traced to a bot detection change affecting those requests. The alert led to a quick fix, aided by tooling like Opentracing.
The jump from pure HTTP simulation to SOAR's full simulation service adds value for customers, since runs can incorporate customer-specific configurations and surface problems unique to their setups. This capability is still in dogfooding internally.
Simulating DoS Attacks
SOAR also helps evaluate mitigation effectiveness and tune thresholds. For Magic Transit's ACK flood mitigation system flowtrackd, performance simulation cases were onboarded alongside tunable ACK floods. Combined with customer-specific configuration, this allows comparing throughput under varying attack volumes and systematically adjusting mitigation thresholds. The attack simulation system also replays historical attacks, guarding against regressions that would let known attack types pass through the pipeline to customers.
Outlook
Simulation is not a new technique, but SOAR applies it to reliability, observability, and security at Cloudflare. Adoption is early, and the expectation is that full use of simulations will push quality of service higher.
