The problem with 80,000-line YAML files

Cloudflare's infrastructure used to be managed the old-fashioned way: massive, source-controlled YAML files — sometimes 80,000 lines long — that defined the workload for every node in its control plane data centers. Each entry described a node, its name, its rack, and the roles that configuration management software would read to assign workloads.

That approach became unmanageable. Engineers updating workload assignments had to touch multiple files and edit hundreds of lines by hand. The data wasn't easily accessible to other systems, either. A centralized, strongly-typed system was needed.

The result is Zinc, an in-house system written in Rust that serves as the source of truth for logical and physical infrastructure assets at Cloudflare. Zinc provides first-class data models for servers, network devices, data centers, and other assets, and exposes that data through APIs, a web UI, and a command-line tool.

Zinc's feature set has grown around operational needs:

  • Workload management — Zinc replaces the role of those YAML files, managing metadata like a node's associated cluster or designated kernel version.
  • Least-privilege user accounts — Through Cloudflare Access, every employee has an individual account with permissions scoped to their job role, preventing unauthorized access to sensitive asset information.
  • Change requests — Similar to pull requests, changes to nodes and their workloads require approval from the team that manages the workload before being applied.
  • Node reservations — Engineers can reserve specific hardware for testing or future capacity, with visibility into what a node is being used for even when it's not in production.
  • Node metadata — Hardware information such as CPU, memory, generation, chassis, power, and networking configuration is collected from other Cloudflare subsystems and unified in Zinc's APIs and interfaces.

Two systems, complementary roles

Zinc doesn't work alone. Cloudflare runs two critical infrastructure systems: Zinc handles logical infrastructure and operational configuration, while Netbox acts as the Datacenter Inventory Management System (DCIM), storing details such as hardware specifications, serial numbers, cable diagrams, and rack layouts.

Zinc imports that physical data from Netbox and builds higher-level abstractions on top, so other Cloudflare systems can depend on Zinc without needing to understand the specifics of datacenter infrastructure.

Automating the server lifecycle

With growth came pressure on manual processes around provisioning, maintenance, repairs, and diagnostics. Zinc became the natural home for workflow automations targeting each touchpoint in a server's lifecycle.

Repairs. Hardware failures are common at Cloudflare's scale, and returning failed hardware to production quickly matters for capacity and cost. Previously, engineers manually collected failure details and filed tickets for data center technicians. Zinc automates that process entirely: with a few clicks — or driven by other automation — an accurate service ticket is created, enabling technicians to make repairs and get servers back into production faster. Future work will push automation down to the individual hardware component level.

Diagnostics. Zinc integrates directly with INAT (Integrated Node Acceptance Tests), a diagnostic service used to identify hardware issues across the fleet. Zinc runs acceptance tests before, during, and after server provisioning, and can execute them ad hoc to determine machine health. Once diagnostics complete, the Zinc interface shows a report that can be used to assess server health and decide whether action is needed.

Maintenance windows. Zinc is also the origin of the maintenance page on Cloudflare Status. When hardware and network upgrades happen, Zinc orchestrates and tracks the maintenance windows and sends alerts to teams and customers when disruptive — or potentially disruptive — maintenance is scheduled in a region. Automated systems ensure Cloudflare sees no loss in quality of service regardless of where maintenance occurs.

Reboots. Reboots require safety: it would be unsafe to reboot a production Clickhouse node without warning, so dedicated systems ensure traffic is routed away before the reboot. Zinc provides an integration with this reboot management system through its Web UI and CLI, making it easier to queue and execute reboots while adding orchestration logic that leverages Zinc's operational management capabilities.

Engineer-facing tooling

Zinc exposes its data and operations through two interfaces built on the same API: a command-line interface written in Rust and a React single-page web application.

Engineers use the CLI for straightforward queries as well as a more expressive query syntax. For example, querying for Kubernetes workers in a specific cluster while excluding storage and rook nodes is a one-liner. All access and changes at the API layer are properly scoped and authenticated.

The web UI is designed for ease of use, since hundreds of Cloudflare engineers depend on it. Items such as assets in repair, nodes in a specific city or country, or even CPU models are first-class searchable in the interface. The Change Request interface, similar to Git Pull Requests, shows what asset data is changing, who is making the change, and ensures approval by designated staff prior to application in production.

Scaling for what's next

Cloudflare's fleet is growing faster than ever, with expansions to deliver GPUs on the Edge and its R2 storage service. Zinc was built to handle the operational burden that comes with that growth, and it remains the backbone of infrastructure and asset data — with deeper automations and integrations planned to save engineers time and reduce manual errors.