Testing dependencies without mocks
Unit tests verify logic within a service, but they don’t cover the interactions between services. At Facebook’s scale—with a codebase supporting billions of users—integration testing is essential, yet standard frameworks for backend services were lacking. We built our own integration testing infrastructure, designed to be extended, and we’ve used it to implement autonomous testing for stateless backend services.
This work extends an existing integration testing framework that helps engineers write effective tests by providing a code framework, test execution capabilities, and hooks into continuous integration. The framework handles the boilerplate, eliminating common pitfalls like flaky constructs. Three key aspects define integration testing with this framework: environment definition, input specification, and output verification.
The cost of mocks
Unit tests rely heavily on mocks or fakes, replacing external dependencies to avoid side effects. This approach under-approximates the system under test—the mocks only implement a subset of real behaviors—so some bugs slip through. Maintaining mocks also requires concentrated engineering effort.
Integration tests reduce this burden by running one or more unmodified services. This means the code exercised in tests is the same code that runs in production, making the test results more representative. A test environment that serves no production requests is created within the existing containerization and routing infrastructure. Some tests can be granted read-only access to production assets; others are isolated from production systems entirely.
Service owners can define the test environment boundaries based on which interactions they need to test. Requests from a service are preferentially served within the same environment, but if the needed service is not present, requests can be routed to a mock, an in-memory replica, blocked, or forwarded to production—for example, for read-only operations.
Creating mocks on the fly
In this framework, mocks are built by starting a service with the same interface as the original but a trivial implementation, running in the same address space as the test harness. This design lets the mock and the test logic interact directly. Mock outcomes can be changed while tests run, and each mock method handler is wrapped into a standard Python MagicMock or StrictMock, making it easy to assert on call counts and arguments.
In-memory replicas are available for common dependencies such as storage. The infrastructure also supports blocking connections made from the test environment. Test inputs are provided by the test harness—a program executing alongside the services under test—that interacts with the services via RPC or changes the environment, like applying new configurations or shutting down a service replica. Mocks can also serve as inputs, sending responses that affect the service under test, including simulated dependency failures thrown as exceptions.
Outputs and extensions
Most test oracles are custom assertions about the externally visible behavior of the service—RPC responses, mock call arguments, or data written to ephemeral test databases. The infrastructure also blocks on generic failures, such as crashes, sanitizer-detected errors, and service health problems from monitoring. Tests check these oracles as usual.
Extension points built into the framework allow teams to address common patterns in their own services, including disaster-readiness tests that bootstrap foundational infrastructure like ZooKeeper from scratch. The framework’s flexibility enables autonomous testing to anticipate developer needs:
Autonomous testing—using fuzz-like input generation over the framework—lets developers focus on brainstorming useful autonomous tests. By defining a simple requirement—such as the maximum number of service inputs in a test—an autonomous automation process generates and runs thousands of unique integration tests at a time. Tests that fail show up consistently and reliably, avoiding flaky behavior. This gives developers a high-signal yield of real failures found at scale.
Safe test environments without human intervention
Integration testing at scale runs into a fundamental challenge: how do you exercise a service without letting its test traffic leak into production systems? For many services, the answer is to mirror production as closely as possible while carefully controlling the boundaries. The testing infrastructure builds an environment by inspecting the service's Twine definition — the standard format Facebook services use with the cluster management system — then programmatically modifies it for testing. Those modifications include isolating the instance from production, lowering resource requirements to save capacity, and other small adjustments. The environment itself is also validated before use, with sanity checks that flag cases needing human attention, such as a service requiring special hardware unavailable in the default test pool.
Isolation turns out to be the most sensitive part. The infrastructure decides what inputs to feed a test, so the test must have zero side effects regardless of those inputs. A concrete failure illustrates the stakes: in one incident, test data about API failures reached the monitoring infrastructure, which mistakenly attributed the failures to production and raised spurious alarms.
Complete isolation, however, frequently produces broken tests — services in a fully sealed environment often fail for reasons unrelated to the code under test. The practical approach sits between those extremes:
- Known read-only traffic is allowed through.
- Service owners can set an allowlist for safe destinations.
- All standard RPC traffic is rerouted to a universal mock that impersonates any service and returns falsy values.
- All other network requests are blocked.
With this layered policy, one-third of services run in a safe test environment with no human involvement at all.
The implementation blends two isolation mechanisms. Application-level isolation handles RPC calls and can block connections based on the specific API being invoked. Network-level isolation works at the IP:port granularity and is typically applied to well-known services like DNS. The isolation system can also make decisions based on the code initiating a connection, not just its destination — useful because some code safely uses APIs that look dangerous on their face. The system identifies the caller by inspecting the stack trace at runtime.
Between two implementation candidates, BPF and LD_PRELOAD, the latter won out for its flexibility. The preload logic fetches a per-service isolation configuration from the configuration management system, then intercepts calls to the libc connect, sendto, sendmsg, and sendmmsg functions to enforce it.
Automating input generation
Manual test writing only gets you so far. To push automation further, the team looked at fuzzing — a natural fit because its input generation is automatic while its dynamic nature slots into the classic testing paradigm. Fuzzing in the traditional sense requires three manual steps: carving out a self-contained unit of code as the target, writing a harness that shapes random bytes into the types the code expects, and ensuring the generated values respect whatever constraints the code implicitly requires. That last step is easy to underestimate. Fuzzing strlen, for example, requires every input to be a valid pointer to a NULL-terminated string; violate that and you get crashes that indicate nothing about real bugs.
Integration testing eliminates those manual steps when the service is expressed through Thrift, Facebook's RPC framework. Its interface definition language exposes the full API contract programmatically, letting code enumerate APIs and recursively inspect argument types and attributes like requiredness. That capability serves two purposes: constructing inputs for the service under test, and mocking the service's dependencies with automatically generated default responses. No manual carving out is needed — the whole service is the target. And because a service typically has no implicit constraints on data arriving over the network, crashes found this way point to real defects rather than harness artifacts.
The simplest input strategy — randomly choosing values matching each type — still delivers value. It establishes a testing baseline and surfaces corner cases engineers tend to miss in hand-written tests. Supplementing with manually chosen extreme values per type, such as MAX_INT for integers, strengthens the approach further.
Random testing has a well-known blind spot, though: when valid input depends on a checksum or other cross-field constraint, naive fuzzing can't produce input that passes validation and therefore never exercises logic beyond it. To push past that limit, the infrastructure records a small fraction of production requests, sanitizes them, and offers them as test inputs. Mutating these recorded requests preserves enough of their original valid structure to exercise deep code paths while still providing randomness to probe edge cases on those paths. At the opposite extreme, replaying recorded requests unmodified serves as an automatic canary: it verifies a new service version can handle the previous version's real traffic without behaving abnormally — all without separate infrastructure or any impact on production.
Beyond crashes: spotting vulnerabilities
Crashes and sanitizer output like ASAN are not the only signals. The testing infrastructure also flags undeclared exceptions and scans logs for suspicious messages. Two recurring exception patterns stand out. MySQL ProgrammingError indicates the fuzzer managed to alter a SQL query through unexpected API arguments — evidence of SQL injection. Similarly, Python SyntaxError appears when fuzzed input reaches a string passed to eval, pointing toward possible arbitrary code execution.
Auto-generated tests need no special oracle setup beyond these: crashes, sanitizer findings, and behavioral anomalies. API contract violations turn out to be a genuinely useful signal, and notably easier to debug than crashes.
Rollout and findings
Deployment followed two phases. First, autonomous integration tests ran in the background on as many services as possible, without requiring owner participation. That phase was meant to surface improvement opportunities and refine issue reporting. Second, service owners were invited to opt in to running the tests automatically in their deployment pipelines. The opt-in model exists because a test failure demands immediate action to unblock deployment; running the test by default would impose that burden unasked. The rollout is now transitioning from phase one to phase two.
In phase one, the combination of the isolation layer and fuzz integration testing allowed safe, fully automatic fuzz testing of roughly one-third of Facebook's Thrift services. Those runs identified more than 1,000 bugs, each filed as a report to the owning team. The remaining two-thirds of services were excluded for reasons such as nonstandard setup, strict permissions preventing reuse of production artifacts, or failure under the enforced isolation policy.
Several lessons emerged. First, the experience highlighted real opportunities to refine test environment isolation, pushing toward fine-grained labeling of read-only APIs and first-class abstractions that allow composing and reusing test environments. Second, bug reporting matters enormously: integration failures are harder to debug than unit test failures, so engineers benefit from maximum context about the service and its libraries, not just a stack trace. Third, purely random inputs make bugs harder to reason about. Shifting toward recorded traffic and well-formed inputs produces failures engineers can actually understand. The experience also pushed back against the notion that services may legitimately break their API contract under random input — thorough input validation is the correct posture. Finally, coverage visibility is essential. Bugs found so far have served as the primary effectiveness metric, and whole-service coverage measurement is now rolling out to give owners insight into under-tested areas and guide further infrastructure improvements.
