Automating Datacenter Operations at Dropbox

Validating infrastructure before it hits production

Dropbox runs its own infrastructure, which means every new server and network switch must be installed, checked, and configured before it can carry production traffic. Historically, this process involved a lot of manual work. The Pirlo system was built to automate most of that effort. It validates and configures network switches, confirms that servers are reliable, and manages the job queue that ties both workflows together.

Pirlo is split into two main components: the TOR Starter handles switch provisioning and validation, while the Server Validation component checks servers before they enter production. Both feed into a shared job queue called ClusterOps. In Dropbox's terminology, users are people, while workers are technologies that execute tasks.

A purpose-built job queue

At its core, Pirlo relies on a distributed job queue backed by MySQL and built in-house on top of Dropbox's existing primitives, such as gRPC, service discovery, and managed MySQL clusters. The team considered third-party tools like Celery but decided they offered more complexity than the use case required. Building on internal components allowed a small group of SREs to both develop and operate the service while keeping the design flexible.

The ClusterOps queue is intentionally generic. It provides four reusable pieces:

• A basic Job table.
• A database utility built on the SQLAlchemy toolkit.
• A queue manager thread interface.
• A worker thread interface.

TOR Starter: automating switch provisioning

The TOR Starter is the Pirlo component responsible for validating and configuring switches. It covers switches in datacenter server racks, PoP racks, and the various layers of the datacenter fabric that connect racks within the same facility.

Building the TOR Starter on the ClusterOps queue gives it a standard manager-worker architecture, but the switch job table extends the basic job table, and the queue manager and worker implementations are customized for switch provisioning. The worker contains all switch validation and configuration logic while the manager schedules jobs.

The database also holds tables that provide a full picture of each switch job. As a job runs, clients can query its state and display it in the user interface. Once a job completes, all of its state remains in the database for reporting and analytics. Additional tables track the status of individual switch components, such as uplinks, fans, and power supplies, with every data point linked to the switch job that captured it.

Life cycle of a switch provisioning job

A switch job is created when a user or a switch discovery service makes a gRPC request to a healthy TOR Starter server via service discovery. The switch is checked for provisioning eligibility, and the job is placed in the work queue.

The queue manager processes jobs in FIFO order and assigns an appropriate job handler based on the job type, an enumerated identifier that maps to specific modules in the workflow. These modules provide the worker with all the instructions and tasks needed for provisioning. As the worker executes them, the job moves through a series of phases.

Before validation can begin, the switch needs out-of-band connectivity, provided by devices installed during the network-core build phase of a cluster. Once connectivity is confirmed, the switch must pass a sequence of checks.

The TOR Starter uses a plugin system to separate the code that runs on the switch. Each plugin sets a particular phase when it starts, transitions the job through enumerated phases, and updates the database at each step. Plugins can also throw enumerated failure codes that tell the operator what went wrong and can drive auto-remediation. Each plugin logs events to the database, and those logs are automatically associated with the switch job.

The switch provisioning workflow includes these key plugins:

After the initial plugins pass and each uplink has network connectivity, a dedicated firmware plugin verifies that the switch runs the expected firmware. If an upgrade or downgrade is needed, the switch gets a basic static route and downloads the image from a production server. The TOR Starter then requests a switch configuration from the internal network database and configuration tool developed by Dropbox's Network Reliability Engineering (NRE) team, and copies it onto the switch. Another plugin performs final checks that all routing protocols are functioning. The final plugin reboots the switch and verifies it comes back up with the correct configuration.

Throughout each phase, the TOR Starter captures every command sent to the switch and its output, saving them as log events. Commands vary by switch type and role, but phases and failure codes stay generic across platforms. Most command output is parsed, and failure codes are assigned based on specific output. When a job fails, the failure code indicates the problem, and the collected logs support further diagnosis. If a hardware component fails, the component's table entry is updated to reflect the failed status.

Tracking jobs in the UI

In addition to command-line clients, Pirlo provides a web interface for the TOR Starter. For each switch job, the UI shows a holistic view of captured data alongside a running list of events. Users can watch jobs in near-real time as the worker executes each plugin and transitions through phases.

On success, the UI shows each phase completing cleanly. On failure, it highlights the problem. If the failure code is PSU_FAILURE, for example, the UI identifies Power Supply 1 in red so the operator knows exactly which component needs attention.

Validating Servers With Pirlo

Pirlo Server Validation handles both provisioning of new servers and the re-validation of repaired machines. Every new server entering a Dropbox data center runs through Pirlo as its first step, and repaired servers must pass validation before returning to production.

The service reuses the same ClusterOps queue infrastructure as switch provisioning, with a custom table extending the standard job table to represent server jobs. A dedicated queue manager schedules those jobs, while the worker implements the full server validation logic.

Booting Into Hotdog

Validation uses a PXE-booted RAM disk image called Hotdog, built with Debirf. The image is a minimal Ubuntu augmented with vendor tooling for RAID, BIOS, and BMC management, plus benchmarking and stress utilities and Dropbox code that communicates with Pirlo for inventory checks, stress application, and recording results.

Running entirely from main memory means validation never depends on a functional storage subsystem. After the network boot, startup scripts syncing additional tooling over rsync, which keeps that tooling updated independently of the base image. While Hotdog exists primarily for Pirlo automation, operators can also boot a machine into it manually for interactive debugging of stubborn systems.

Server Job Design

Each server job’s attributes and related tables capture its full state. Clients can query that state live during execution and display it in the UI. Completed jobs retain all state in the database for reporting and analytics, including event data with logs and structured text.

Benchmark results for server components also live in the database for statistical analysis. A job table row links the server in the asset database, and every job triggers a snapshot of server inventory sent there. Job state records any phase that runs, the results, and a snapshot of the server's component configuration.

Job Lifecycle and Types

For the server to be validated it must first be able to boot into Hotdog. From there, a series of plugins must succeed in the order defined by the job handler for that job type. The handler maps each job type to a specific plugin sequence, enabling a variety of validation jobs that run subsets or full suites of tests. Every job carries enumerated phases and failure codes, with plugins logging data to the database as they go.

Server jobs come in two types:

Provision jobs are created automatically the first time servers arrive at a data center, right after the corresponding rack finishes its switch provision job. These run configuration and validation steps plus extended burn-in tests that create high-load conditions to exercise hardware components.

Validation jobs run when the Datacenter Operations team moves a server from repair back to repaired status. These comprehensive jobs test and update components, verify the bill of materials, and confirm a successful repair before the server re-enters production.

The earliest plugins check that a host can boot and install dependencies required by later steps. If these fail, as much data as possible is saved from out-of-band interfaces to help operators understand why the machine can’t start.

Plugin Execution

Plugins execute all commands over SSH. Servers under validation intentionally run no Dropbox daemons or service discovery components, so they can’t push data back to the worker; the worker pulls everything it needs.

A generic framework provides long-running command support: the plugin polls a status file on the server to detect timeouts or crashes. Polling runs infrequently, so the overhead has not proven to be a scaling concern.

Verifying the Bill of Materials

Pirlo keeps a detailed spreadsheet of all valid component combinations for each server class. That list gates the make and model of components allowed in the fleet, and the database validates configurations against these hardware-class rules.

A whitelisted memory configuration, for example, must match one of the stored valid layouts exactly before it passes.

BOM verification covers the root and storage disks, memory modules, CPUs, RAID controllers, and network cards. The checks deliver two benefits: no wrong component can be swapped into a machine and still pass validation, and replacement parts from vendors must have been qualified by Hardware Engineering before they can go into production.

Firmware Verification

Hardware Engineering qualifies precise firmware versions for components like the RAID controller, BMC, BIOS, and network card. A plugin runs a Hardware-team-supplied tool that bundles all vendor firmware images and the desired BIOS settings, then performs the upgrade or downgrade to bring every component to the approved revision.

Stress and Benchmarking

The most revealing failures often appear when components are exercised under load rather than sitting idle. Stress tests on memory, CPU, and disk uncover those intermittent component faults in the isolated environment. Benchmarks add another layer: known-bad parts, misconfigured boxes, and strange issues all tend to miss thresholds and get flagged before a server can reach production.

Each test plugin runs based on the server class and installed components, and new tests can be added simply by creating a new plugin. All output is stored either as logs or as structured rows in a dedicated benchmark table. Aggregating that table across jobs lets the team compare any given server against its peers, refine threshold values, and surface odd behavior consistently.

Completion and the UI

As with switches, a server only counts as validated if every plugin in the job type completes successfully. After that, Pirlo moves the server back to production for operating system installation. When the server was under repair, the job also updates the repair ticket with a summary and a direct link to the job UI.

The Pirlo UI mirrors the switch job view: each server job has its phase/log table, live progress while the worker executes, and persistent access to finished jobs in their final state. Structured log entries for things like disks render as tables, while benchmark failures display prominent warnings. Raw benchmark data likewise remains available within the job view.

CPU1:
    DIMM_1: [HMA42GR7MFR4N-TF, M393A2G40DB0-CPB]
    DIMM_3: [HMA42GR7MFR4N-TF, M393A2G40DB0-CPB]
CPU2:
    DIMM_1: [HMA42GR7MFR4N-TF, M393A2G40DB0-CPB]
    DIMM_3: [HMA42GR7MFR4N-TF, M393A2G40DB0-CPB]

Measuring Pirlo’s Impact

Pirlo was built to automate or eliminate manual processes in the datacenter. The team’s core hypothesis was that the system would deliver two main benefits: reduce errors, downtime, and rework from incomplete or erroneous provisioning, and act as a force multiplier for operations engineers by cutting human intervention time.

Before Pirlo, server provisioning and validation required an operations engineer to consult playbooks and subject matter expertise, then use server error logs to prescribe remediation or configuration steps. After remediation, the engineer would send the machine to the re-imaging system to release it back into production. If the actions didn’t cure or properly prepare the server, it bounced back to the engineer for yet another round of triage. These manual workflows consumed significant time and resources and created heavy churn in the re-imaging system.

The team theorized that by combining three capabilities—holistic health checks covering BOM validation, component health and benchmark/stress testing, firmware patching of hardware subcomponents while in the pit-stop, and gating release into re-imaging on a clean bill of health—Pirlo would reduce server failure recidivism. In turn, engineers could take on heavier task loads, knowing they were releasing healthier machines into production.

First-Pass Success Rates

Server failure recidivism is defined as any server that does not obtain a first-pass success. Without Pirlo, those machines required manual diagnosis, validation, and remediation, cycling between the engineer and the re-imaging system until provisioning succeeded. The beta release of Pirlo in January 2018 was intended to raise first-pass success by catching common hardware failures that typically appeared during or right after re-imaging.

As the Pirlo development team incorporated newly discovered systemic issues, software updates, and expanded benchmark and diagnostic suites, the system improved continuously. From January 2018 onward, first-pass success increased by 12.2%.

Effect on Engineering Throughput

That improvement freed up engineers to work on other machines, and they did. Operations engineers increased their output by more than 40% over the same period. Fewer servers competing for attention meant less babysitting while machines sat in re-imaging queues. Pirlo’s automated sequential battery of tests reduced hands-on time while improving diagnostic accuracy, and the shorter time-to-remediation let engineers handle more issues in a single day.

Notably, all of this happened while the internal Dropbox fleet grew by more than 15%. The average percentage of machines in a nonfunctional or repair-requiring state stayed below 0.5%, and the operations engineer headcount remained the same.

Looking Ahead

Pirlo now underpins Dropbox’s physical operations, covering new rack provisioning and server repairs, with subcomponents for both switches and servers under continuous development. Through 2018, the system ran tens of thousands of server and switch jobs and caught severe issues that might otherwise have caused production reliability problems. The accumulated log data from those runs provides a corpus for driving further evolution of Pirlo and its components.