BellJar: A new framework for testing system recoverability at scale

Recoverability as a testable property

Infrastructure that can weather outages—especially those affecting adjacent systems—often suffers from poorly understood fate-sharing between components. Mapping these dependencies and surfacing the side effects of unavailability has traditionally been slow, manual work, compounded by the rarity of large-scale outages and the lack of tooling built for this purpose. The result: teams struggle to reason about how far an outage will spread, which mitigations actually restore service, and whether bootstrapping blockers or circular dependencies lurk beneath the surface. The lower in the stack such outages occur, the wider their blast radius, making purpose-built experimentation essential.

BellJar, a framework developed at Meta, provides a structured way to exercise infrastructure code under worst-case conditions. It offers a controlled environment—indistinguishable from production except that it is vacuum-sealed from all production systems—so teams can prove their infrastructure recovers the way it should.

Coupling by curation, not isolation

At Meta’s scale, infrastructure spans dozens of data centers and millions of machines, rendering manual intervention impossible for even simple tasks like configuration changes. Shared foundational systems solve these operational problems once, benefiting all service owners. But their pervasive reach also means an outage can ripple across vast swaths of software. The conventional wisdom—build code with zero external dependencies for fast recovery—breaks down under the weight of production reality. Infrastructure software still needs to run somewhere, receive upgrades, scale, expose config, store state, and communicate. Rebuilding these capabilities for perfect isolation is impractical; shared infrastructure is often the pragmatic choice, especially where edge cases are hard to get right.

Without deliberate management, however, systems tend to accrete uncurated relationships that create coupling and complexity. Engineers may over-engineer decoupling for some dependencies while unwittingly allowing others to become circular and brittle. Structured controls are necessary to prevent this degradation.

The design constraint of recoverability

Focusing on recoverability offers a tractable way to balance the extremes of duplication and entanglement. Recoverability means having confidence that infrastructure can return to a healthy state promptly after an outage. Because outages can take many forms, BellJar tests against a broad prototypical failure: the categorical unavailability of common infrastructure. This framing simplifies the coupling problem in five ways:

• Practical: Coupling that doesn’t impair recovery is acceptable; total isolation is not required.
• Characterized: Analysis concentrates on the relationships needed to reach minimal health, shrinking the dependency universe under scrutiny.
• Scoped: Individual systems can be examined alongside their adjacent supporting software, treating these as the building blocks of recovery.
• Additive: Instead of enumerating what happens if each of thousands of dependencies is removed, ask what the system truly requires to reach a minimally healthy state.
• Testable: Recovery is a Boolean outcome—success or failure—which can be asserted programmatically in real scenarios.

Experiments in a fault-injected vacuum

Each BellJar instance runs on ephemeral virtual machines (VMs) rotating through a lease-wipe-update cycle. Before handoff to a user, a blanket of faults is injected, blackholing traffic on all network interfaces, including loopback. This fault injection engine supports allowlist-style disruptions at the IP layer, application network layer, or per-process and container level.

Operators customize the experiment by additively leaking production capabilities back into the vacuum by name—DNS, a specific storage cluster, a read-only binary package backend—via a granular checkbox-style allowlist. This defines the recovery conditions: the minimal set of healthy endpooints, daemons, and libraries the operator assumes will be available. Within that scenario, teams can then execute and validate their system’s recovery procedure from bare metal to healthy.

Recipes for recovery

Each BellJar test assembles distinct ingredients, typically varied across multiple scenarios per system:

1. Service under test, such as a container scheduler, config distribution system, coordination back end, or certificate authority—most running in containers.
2. Hardware configuration matching the production deployment.
3. Recovery strategy, the documented runbook for starting from scratch on fresh hardware.
4. Validation criteria proving the recovered system is healthy.
5. Tooling used in the runbook, which operators would rely on during real incidents.
6. Recovery conditions, the exploit allowlist capturing which infrastructure capabilities must remain up for recovery to succeed.

Service owners codify these inputs in a few lines of Python.

Replacing tribal knowledge with code

Before BellJar, untangling recovery order and cyclic dependencies demanded combing through wiki pages, interviewing veteran engineers, and mining commit histories across multiple languages. Now, a BellJar test distills operator assertions into a repeatable, Boolean check. It verifies that the service can be recovered using the specified runbook and tooling under any outage where the defined recovery conditions hold. Tests can run in CI/CD pipelines, asserting recoverability for every release candidate.

Codified recovery tests pay off in two ways. First, they let teams move faster with confidence: changing arcane settings, retry policies, or failover mechanisms no longer threatens disaster recovery posture, as the automated test validates each diff. Second, they drive active decoupling. Onboarding is gradual—teams first define recovery in a fully-provisioned environment, then iteratively strip the allowlist to identify and eliminate surprising dependencies, particularly those posing circular bootstrapping risks. BellJar makes dependency liabilities visible and actionable.

A wider lens on recovery

BellJar’s value grows as more service owners adopt it, because the framework’s broad view exposes cross-system behaviors that are difficult to see in isolation. The project deliberately asks each team the same question — “What does it take to bring your service online on an empty set of hardware?” — and does not require any single owner to understand Meta’s entire dependency graph. But as more teams contribute their answers, those individual first-hop requirements can be assembled into a larger recovery graph that captures dependencies spanning organizational and time-zone boundaries.

This recovery-specific graph is proving distinct from the runtime relationships revealed by distributed tracing. Foundational services such as security, networking, database, and storage each express their portion of the graph as code, which makes the whole structure amenable to programmatic analysis.

Runbooks as executable documentation

Service owners define their recovery strategy as a set of human-invokable steps — essentially the disaster recovery runbook an operator would follow. These steps often rely on powerful toolkits or automation controls, and codifying that human-oriented process inside a test framework may seem odd at first. But human intervention is typically the first response to a major incident. By capturing that runbook in a BellJar test, the framework can automatically generate the human-friendly HTML documentation that operators traditionally maintain by hand on wikis or in offline manuals.

The documentation is regenerated every time the service’s BellJar test passes, eliminating the staleness and toil associated with conventional runbooks.

Federating recovery to service owners

Meta operates scores of infrastructure systems, and no single dedicated team can realistically own recovery readiness for all of them. The company does have specialized teams addressing cross-cutting concerns such as RPC overload protection, safe deployment in the face of unexpected capacity loss, and fault injection tooling. Other domain experts focus on recovering specific portions of the stack, like intra-data-center networking or container management.

However, the goal is not to let service owners treat disaster recovery as someone else’s problem. Those cross-cutting projects succeed because they equip owners to solve specific aspects of readiness, not because they absolve them of responsibility. BellJar extends this model by integrating recovery validation directly into service owners’ development pipelines. The owner designs the recovery strategy, tests it, and receives a failing test signal when changes invalidate that strategy. Embedding recovery in the release cycle pushes developers to design systems that can recover simply and reliably.

Shared tooling for common barriers

Service owners who prioritized disaster readiness previously had to build bespoke recovery tooling on an ad-hoc basis. Teams commonly reinvented similar solutions in silos — for spinning up containers without a healthy orchestrator, distributing failsafe routing information, or fetching binaries from safe storage. BellJar’s universal runbook format makes these common needs visible in one place, allowing the BellJar team to identify shared problems and develop standard solutions. Consulting with dozens of service owners has produced:

• A common toolkit for constructing low-level containers when the normal orchestrator is unavailable.
• Fleetwide emergency binary distribution that bypasses most standard packaging infrastructure.
• Secure access control for emergency tooling that respects existing ACLs.
• Centralized collection and monitoring of recovery inputs such as metadata and container specifications.

These well-tested shared tools get first-class support under dedicated team ownership.

Recurring recovery dependencies

Analyzing recovery requirements across many systems has surfaced patterns about what foundational services and design decisions really matter when the worst happens.

Security infrastructure comes first

Access controls and TLS govern every action operators and automated systems can take, in steady state and during recovery alike. Certificate authorities are therefore central to any recovery strategy. When every container requires an x.509 certificate from a trusted issuer, avoiding dependency cycles among containerized security services becomes especially difficult. These systems need early investment before robust recovery of the rest of the stack is possible.

DNS health is a hidden assumption

Even with custom service discovery, DNS is woven through tooling everywhere. Because it is typically robust, engineers treat it almost like a local resource — as reliable as a filesystem — and forget it is a network-accessed system. Viewing DNS as fallible and remote requires deliberate recalibration. That discipline is especially important in an IPv6 environment where 128-bit addresses cannot be committed to a sticky note. For select systems that power discovery, Meta has moved to routing protocols that stabilize well-known IP addresses, and is retrofitting much of its tooling to accept IP addresses in an emergency.

Hardening can hide worst-case failures

Years of caching and fallback logic make systems very difficult to break fully, but those availability trade-offs obscure how catastrophic failures will manifest. Service owners often see production without any configuration or discovery caching for the first time in a BellJar environment. The framework therefore devotes considerable effort to ensuring tests actually break things in the worst possible way.

Configuration also needs a bootstrap

Everything at Meta’s scale is configuration-as-code — feature knobs, killswitches, RPC protocol features, security ACLs, hardware package installs, and even the configuration distribution systems themselves. Emergency CLI tooling and fleetwide daemons follow the same model. When all of that depends on a common configuration distribution system, teams need robust emergency delivery, automatic fallback to prepackaged values, and diverse contingency plans for when the configuration underpinning the configuration system itself is broken.

Dependency creep demands allowlists

Low-level services increasingly depend on libraries that evolve faster than any single team can scrutinize line-by-line. Those libraries are vectors for unintended dependencies, and at Meta’s codebase scale, blocklist-style fault injection is a losing game: you cannot enumerate everything in production to know what to test. Unwanted dependencies have surfaced in unexpected places — a Python2 to Python3 upgrade tying recovery tooling to esoteric services, authorization checks creating hard dependencies between emergency utilities and a web front end, and logging or tracing libraries unexpectedly blocking container tooling with SIGABRT. Allowlist-style BellJar tests catch such dependencies automatically, revealing connections to systems nobody knew were involved.

Expanding beyond services

BellJar’s constrained environment is also being applied to code that does not look like a typical container or network service. Common internal libraries for RocksDB builds and distributed tracing now use BellJar to assert they can initialize and serve requests when none of Meta’s upstream systems are online. Sidecar binaries running on millions of machines — handling data distribution, host management, debugging, and traffic shaping — are vetted with tightly restricted allowlists. Each new daemon build must demonstrate it can auto-recover even when the upstream control planes it normally depends on go dark.