Automating data center expansions with Airflow

Airflow automation trims hands-on toil from Cloudflare’s server rollouts

Keeping Cloudflare’s network current means more than opening new sites. Existing data centers are continuously refreshed with newer server generations, and each refresh used to demand hours of careful, manual execution. Operators from the Data Center and Infrastructure Operations, Network Operations, and Site Reliability Engineering teams worked from a lengthy standard operating procedure (SOP), copying command snippets from the document into terminals.

That approach stops scaling once a network reaches a certain size. Cloudflare’s answer was a self-built Provisioning-as-a-Service (PraaS) platform that replaces those manual steps with API calls and cut time spent on routine operational work by 90%.

Replacing doc-following with DAG tasks

PraaS is built on Apache Airflow, the open-source workflow platform organized around directed acyclic graphs (DAGs). In PraaS, every step of the old SOP becomes a task inside one of those DAGs. Most tasks are API calls to Salt, the configuration management system Cloudflare uses for servers, switches, and routers. Others query Prometheus and Thanos for monitoring data, post to Google Chat, open JIRA tickets, or reach internal systems.

The shift is obvious in how a routine action changed shape. The SOP told an SRE to log into a remote system, paste a command with a placeholder, find the correct router name, and execute. That procedure is now a single DAG task called enable_anycast:

enable_anycast = builder.wrap_class(AsyncSaltAPIOperator)(
             task_id='enable_anycast',
             target='{{ params.netops }}',
             function='cmd.run',
             fun_kwargs={'cmd': 'salt {{ get_router(params.colo_name) }} '
                         'anycast.enable --out=json --out-indent=-1'},
             salt_conn_id='salt_api',
             trigger_rule='one_success')

Airflow’s extensibility made it the right framework for this transformation. Each task is an instance of an Operator, and Cloudflare engineers wrote custom operators where needed. The AsyncSaltAPIOperator shown above is one example; SRE teams have produced operators for Salt, Prometheus, Bitbucket, Google Chat, JIRA, and PagerDuty.

Tasks that do more than run a command

The new automated steps bundle behavior far beyond the old copy-paste instruction:

• Failure handling. Tasks retry automatically up to a configured limit. Retry policies vary, and some tasks are explicitly set to never retry — the sensible choice when a retry is impractical or the failure condition is unlikely to change.
• Logging. Each task writes a detailed execution log, aimed at making audits and troubleshooting straightforward.
• Notifications. Tasks report DAG name, task name, state, attempt count, and log links. Failures trigger extra context, such as retry counts, wiki links, and Grafana dashboards. When the issue is critical enough, the task can page the on-call provisioning engineer.
• Jinja templating and macros. Templating lets otherwise static fields become dynamic, and macros feed runtime parameters into task instances. Macros evaluate while the task runs, so the values reflect the current state of the workflow.

Gating progress on preconditions and people

Some SOP steps exist purely to confirm that prerequisites are met. In Airflow, that role belongs to sensors. PraaS uses sensors to block a task or an entire DAG until a dependency finishes successfully — for example, waiting for all nodes to resolve to the correct DNS records:

verify_node_dns = builder.wrap_class(DNSSensor)(
            task_id='verify_node_dns',
            zone=domain,
            nodes_from='{{ to_json(run_ctx.globals.import_nodes_via_mpl) }}',
            timeout=60 * 30,
            poke_interval=60 * 10,
	mode='reschedule')

Human sign-off is handled the same way. A sensor can send notifications to an operator and pause progress until a Change Request ticket is provided and checked:

verify_jira_input = builder.wrap_class(InputSensor)(
            task_id='verify_jira_input',
            var_key='jira',
            prompt='Please provide the Change Request ticket.',
            notify=True,
            require_human=True)

Another sensor waits until a Cloudflare engineer has deployed the zone as part of DNS infrastructure work. To let an operator inject input into a running expansion, PraaS uses a dedicated DAG Manager. The operator triggers it with a JSON configuration, and the DAG Manager submits that input back to the expansion workflow.

Dependency management across tasks

Defining the order of work is the second half of building a workflow. Airflow’s bit-shift operators make chaining straightforward:

verify_cr >> parse_cr >> [execute_offline, execute_online]
        execute_online >> silence_highstate_runner >> silence_metals >> \
            disable_highstate_runner

Lists of tasks can be specified as upstream dependencies as well:

change_metal_status >>  [wait_for_change_metal_status, verify_zone_update] >> \
evaluate_ecmp_management

Downstream tasks run only when the upstream succeeds by default. For more nuanced cases, every operator accepts a trigger_rule argument that tells the scheduler when to fire. PraaS relies heavily on one_success, which starts a task as soon as a single parent succeeds rather than making it wait on all parents.

Branching and reusable DAG patterns

Real-world provisioning includes conditional paths, so PraaS uses Airbnb’s BranchPythonOperator to route a workflow based on conditions that emerge from previous tasks.

Rather than concentrating all logic into a single massive DAG, Cloudflare built separable, fully reusable DAGs that can be triggered manually or programmatically. Triggering DAGs from other DAGs proved a useful pattern for handling complex flows, including those that need loops. A DAG is acyclic by definition, but a “helper” DAG can respawn itself when work must iterate, effectively performing cycles within the broader system.

That design choice let expansions decompose cleanly. Each data center DAG reuses others by triggering them, and inter-DAG dependencies enable workflows far more complex than any single graph could express.

Scaling DAGs to hundreds of sites

An expansion happens in two phases. In phase 1, new servers power on, boot a custom Linux kernel, and start provisioning. In phase 2, those servers are enabled in the cluster and receive production traffic.

Writing two DAGs per data center would mean maintaining 400 files across more than 200 sites. Parameterizing the DAGs was another option, but tracking progress across runs would confuse operators.

Instead, PraaS follows the DRY (Don’t Repeat Yourself) principle and borrows from the Factory Method design pattern. The phase 1 and phase 2 DAG code dynamically generates identical DAGs for each data center from one shared codebase. New sites require no code changes — a DAG is produced automatically on top of Airflow’s built-in web UI, which hides the underlying complexity from operators.

SOPs aren’t dead — they’re code

Automation didn’t eliminate the SOP document. Components fail, and when a task does, the manual process becomes the fallback that keeps provisioning and expansion moving. PraaS therefore enforces an SOP-as-Code practice: every DAG task has a matching manual step that an SRE can perform by hand, and any SOP change arrives as a corresponding pull request in the codebase.

Next for PraaS

Onboarding additional provisioning activities into PraaS is already underway — decommissioning is the current focus. For expansions, the long-term goal is a fully autonomous system that detects newly racked servers in edge data centers and triggers provisioning with no human in the loop.