Failures as a given, remediation as a service
At Cloudflare’s scale, component failure is an accepted constant. What matters is the speed and consistency of the response. Hardware errors, overheating, and other lower-priority faults happen often enough that manual, runbook-driven remediation becomes a significant source of operational toil. We needed a system that could automatically detect these conditions and trigger recovery actions without requiring a human in the loop for every single incident.
Our earlier attempts to automate this process resulted in a patchwork of ad-hoc shell scripts, cron jobs, and bespoke services. Each solution worked in isolation, but together they created duplication, tight coupling, and a lack of shared context. This made it difficult to scale and slowed down our response times, increasing Mean-Time-To-Resolve (MTTR) and risking user-facing errors.
Our goal was to build a centralized, self-service platform for auto-remediation. It would provide a single, generic interface for engineering teams across the company (not just SRE) to define and trigger recovery workflows for machines, services, or dependencies.
We based this platform on Temporal, an open-source workflow engine. Temporal’s durable execution model was a logical fit: it gracefully handles network outages and transient service failures, providing reliability guarantees by default. This freed us to focus on building the orchestration and control plane around the workflow engine itself.
Platform architecture
Our automatic remediation system is a way to schedule tasks across our global network. It is built as a single Go binary that can run in one of three roles, each deployed an apt package or a container.
Coordinator: the gatekeeper
The coordinator is the primary entry point for scheduling workflows from other internal services. Its core duties are authorisation, task routing, and safety constraint enforcement. Clients authenticate via mTLS, and an Access Control List (ACL) determines if a requested workflow execution is permitted.
server_config {
enable_tls = true
[...]
route_rule {
name = "global_get"
method = "GET"
route_patterns = ["/*"]
uris = ["spiffe://example.com/worker-admin"]
}
route_rule {
name = "global_post"
method = "POST"
route_patterns = ["/*"]
uris = ["spiffe://example.com/worker-admin"]
allow_public = true
}
route_rule {
name = "public_access"
method = "GET"
route_patterns = ["/metrics"]
uris = []
allow_public = true
skip_log_match = true
}
}
Workflows are defined in an HCL configuration file that declares where a task may run and any safety constraints. For instance, a workflow might be restricted to a specific node type, or have a limit on how many times it can execute concurrently. A high execution count is often a signal of a systemic issue that requires human investigation. The coordinator checks the current state of executions via the Temporal Visibility API to enforce these constraints.
task_queue_target = "<target>"
# The following entries will ensure that
# 1. This workflow is not run at the same time in a 15m window.
# 2. This workflow will not run more than once an hour.
# 3. This workflow will not run more than 3 times in one day.
#
constraint {
kind = "concurency"
value = "1"
period = "15m"
}
constraint {
kind = "maxExecution"
value = "1"
period = "1h"
}
constraint {
kind = "maxExecution"
value = "3"
period = "24h"
is_global = true
}
Task routing
A key feature of using a central Temporal cluster is task routing, which enables us to schedule workflows on any server running a Temporal Worker. To maximize efficiency, we segment workers into three primary task queues:
- General queue – executes tasks that can run on any worker in the datacenter.
- Node type queue – for tasks that can only be executed by a specific type of node, such as a database.
- Individual node queue – targets a specific node for a task.
This routing gives us granular control over where and how tasks run. We can direct tasks to datacenters with lower latency to an external resource or to hardware with better performance characteristics. A known drawback is that every workflow/Activity must be registered to its target task queue, but this is a common failure condition we can catch with proper testing.
Triggering self-healing actions
With the workflow engine and scheduling in place, the next challenge was defining the triggers. We intentionally built this to be flexible: any authorized system that can accurately detect a failure condition can trigger a workflow. We have implemented a daemon that runs locally in datacenters to poll a signal source, and currently find Prometheus to be most useful because it contains both service-level and hardware metrics. We are also looking into event-based triggers that would eliminate the polling overhead. Additionally, our existing internal detection systems can now automatically react to widespread customer-facing issues, creating an automatic feedback loop.
Since the daemon runs locally, it can also trigger workflows from a local coordinator, which is ideal for data centers with degraded performance—it eliminates the round-trip to a central coordinator.
Testing and deployment strategy
Temporal’s native test suite supports unit, integration, and end-to-end testing, which we use to prevent regressions. While valuable, we found that tests alone were insufficient. Engineers need an environment that mirrors production conditions. We configured our staging environments to point to a separate Temporal cluster, allowing quick and realistic experimentation** before a full release to production. This has been critical for catching simple configuration errors that could otherwise interrupt service.
Production outcomes and direction
The system is now live and is used to automatically respond to server-specific errors and unrecoverable failures by removing the errant single points of failure from production. Beyond resilience, we have used this platform to reduce internal operational toil—such as automating the process of wiping and resetting servers after experiments and testing pull requests on target machines. A unified platform maintained by multiple SRE teams has enabled us to iterate faster and get on a path towards eliminating the human bottleneck in our scaling efforts.
Roadmap and broader impact
The immediate goal for this automation layer is twofold: improve platform reliability for customers and cut down on operational toil so engineers can focus on larger-scale problems. The team also plans to adopt more Temporal capabilities, including Workflow Versioning, which ensures triggered workflows execute the expected version, simplifying changes to existing workflows.
Beyond that, Cloudflare is interested in how other organizations approach durable execution platforms like Temporal and in general strategies for reducing toil. The company welcomes discussion on this topic through the Cloudflare Community.
