How we use HashiCorp Nomad

Reliability models for edge services

Every Cloudflare data center runs two broad categories of software: customer-facing services that handle traffic directly, and management services that keep the data center operational. These categories have very different reliability requirements, and until recently we treated them with very different deployment strategies.

Customer-facing services — caching, WAF, DDoS protection, rate-limiting, load-balancing — are deployed on every machine in a data center. This model is simple and scales naturally: adding machines adds capacity. Our dynamic load-balancing system, Unimog, runs on each machine and continuously redistributes traffic based on resource usage and service health, keeping utilization roughly even across all machines and providing resilience to individual machine failures.

Management services are a different problem. These services don't sit in the request path, but the data center may need only one or a few instances of each. We had two traditional options, both suboptimal:

• Deploy to all machines: reliable but wasteful, consuming resources that could serve customer traffic.
• Deploy to a static set of machines: cost-effective but fragile — if those machines fail, the service may go down.

When management services become unavailable, we sometimes have to disable the entire data center while recovering them. The network handles this gracefully — traffic is automatically re-routed to the next nearest data center — but disabling a data center means users are served from further away, which we want to minimize.

What we needed was a dynamic task scheduler: a system that guarantees a certain number of instances of a service run in each data center, without caring which physical machine runs them.

Choosing Nomad

We evaluated candidates against our requirements and selected HashiCorp Nomad. The key factors:

• It directly satisfies our core need: reliably running a small number of instances of a binary with resource isolation in each data center.
• It has few dependencies and integrates cleanly with Consul, which we already run in every data center for service discovery and distributed key-value storage.
• It is a lightweight single Go binary, easy to deploy and provision — important when you're rolling out as many clusters as we have data centers.
• Its modular task driver architecture supports not only containers but also plain binaries and custom drivers.
• It is open source, written in Go, and has a responsive maintainer community. We have deep Go experience in-house.

Kubernetes was considered but did not fit these requirements as well for our use case.

Deployment architecture

Nomad has two components. Nomad Servers form the scheduling cluster; we run five per data center for failure tolerance. Nomad Clients execute tasks and run on every machine in every data center.

We placed the five Nomad Servers to maximize failure-domain diversity:

• across different interconnected physical data centers that form a single location;
• across different racks connected to different switches;
• across different chassis (our edge hardware is mostly multi-node chassis, each holding four servers).

We also added logic to our configuration management tool to maintain a consistent number of Nomad Servers as servers are expanded and decommissioned nearly daily. The logic works by redistributing the Nomad Server role to a new list of machines, then running Nomad Server on the new machines before disabling it on the old ones. Because expansions and decommissions typically affect a subset of racks at a time, and the role assignment provides rack diversity, the cluster quorum is preserved throughout.

Job files and constraints

Nomad job files are templated and checked into git. Our configuration management tool schedules those jobs in every data center; from there, Nomad maintains them continuously.

Each Nomad Client exposes rack metadata, which lets us constrain jobs so instances of a service land in different racks — tying them to different failure domains. A rack failure then can't take down all instances of a service. We use this job file constraint:

constraint {
  attribute = "${meta.rack}"
  operator  = "distinct_property"
}

Service discovery and observability

Nomad's Consul integration dynamically registers jobs in the Consul Service Catalog, so we can discover where a service runs in each data center by querying Consul. With the Consul DNS interface enabled, we can also target Nomad services via DNS lookups.

Operating clusters in every data center requires solid observability. We scrape Nomad Servers and Clients with Prometheus, use Alertmanager for alerts on key metrics, and built Grafana dashboards for visibility into each cluster.

Prometheus discovers services running on Nomad by querying the Consul Service Directory, using this configuration to scrape their metrics periodically:

- consul_sd_configs:
  - server: localhost:8500
  job_name: management_service_via_consul
  relabel_configs:
  - action: keep
    regex: management-service
    source_labels:
    - __meta_consul_service

Those metrics feed our Grafana dashboards and alert rules for Nomad-scheduled services.

Access to Nomad API endpoints is restricted with mutual TLS authentication. We generate client certificates for each entity that interacts with Nomad, so only entities with valid certificates can schedule jobs or perform CLI operations.

Challenges encountered

Initramfs and pivot_root

When we started using the exec driver to run binaries isolated in a chroot, we found that our stateless root partition running on initramfs was unsupported. Tasks would fail to start with a pivot_root invalid argument error.

Feb 12 19:49:03 machine nomad-client[258433]: 2020-02-12T19:49:03.332Z [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=fa202-63b-33f-924-42cbd5 task=server error="failed to launch command with executor: rpc error: code = Unknown desc = container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:109: jailing process inside rootfs caused \\\"pivot_root invalid argument\\\"\""

We filed a GitHub issue and submitted a workaround pull request, which was reviewed and merged upstream. In parallel, we modified our boot process to enable pivot_root for stronger isolation, and other team members proposed a kernel patch to make this easier in the future.

Containing resource usage

Tasks running on Nomad share a machine with other services, so we needed strict resource containment. Disk is a shared resource, so we isolated Nomad's data directory to a dedicated, fixed-size mount point. Nomad doesn't yet support limiting disk bandwidth or IOPS out of the box.

Nomad job files have a resources section where memory (in MB) and CPU (in MHz) can be limited:

resources {
  memory = 2000
  cpu = 500
}

These limits are enforced with cgroups. Our testing shows memory limits work as expected, but CPU limits are soft — they aren't enforced while the host has available CPU.

Workload unpredictability

Today, all machines run the same customer-facing workload, and Unimog keeps resource usage nearly identical across machines. Scheduling individual jobs dynamically challenges that homogeneity. Batch-style jobs with spiky resource usage can create uneven load, and we need to watch for feedback loops where Unimog reacts to batch bursts.

As we onboard more services, we will address this by constraining job resource spikes and ensuring Unimog handles batch workloads without positive feedback.

Running critical services on Nomad

With Nomad active across all of our data centers, we began onboarding essential management services to improve their resilience. The first candidate was our reboot and maintenance management service.

Reboot and maintenance management service

Previously, each data center ran this service—which handles unattended rolling reboots and machine maintenance—on a single designated machine. That setup left it vulnerable to individual machine failures; if that node went down, machines couldn't automatically re-enable themselves after a reboot. Moving this service onto Nomad was an obvious first step toward hardening it.

Today, we're guaranteed that this service is always running in every data center, independent of individual machine failures. Rather than relying on a hardcoded address to reach the service, other machines now query Consul DNS and dynamically resolve where the service is running before interacting with it.

The reliability gain here is substantial, and it sets the pattern for other management services we plan to migrate in the coming months. We're looking forward to rolling those out.