Containerizing ZooKeeper with Twine: Powering container orchestration from within

When the infrastructure underneath needs a lift

Apache ZooKeeper has long been the quiet foundation beneath Facebook’s container infrastructure. Twine, the internal orchestrator, leans on ZooKeeper for leader election, fencing, distributed locking, and membership management across the services it runs. For years, that relationship went one way: ZooKeeper powered Twine, but never the other way around. Kept off the container platform to avoid circular dependencies, ZooKeeper clusters lived on bare metal, managed by bespoke tooling that duplicated what Twine already provided. Earlier this year, that changed. The final ZooKeeper cluster moved off bare metal and onto Twine’s managed ecosystem, putting the orchestrator in charge of the very service that keeps it alive.

That migration was not merely a packaging exercise. It meant reconsidering what a low-dependency service really needs — and giving up hard-won operational control in exchange for automation.

Why ZooKeeper stayed out of the pool

A ZooKeeper ensemble is typically five participants: one leader replicating to four followers. Losing three simultaneously means irreversible data loss, so the system is built around guarantees of linearizability and durability for the thousands of use cases it serves — machine learning pipelines, Unicorn, LogDevice, and the foundational infrastructure that runs Facebook’s containers. Twine schedulers, service discovery, Configerator for configuration distribution, and fbpkg for package replication all depend on ZooKeeper.

Because of that central role, ZooKeeper was managed with its own custom infrastructure to avoid circular dependencies and ensure recovery during disasters. But that isolation created its own problems. Deployment lacked the hardened health checks and rollback capabilities of Conveyor, the standard release tool. Failure-domain spread, normally the scheduler’s job, had to be handled with a custom solution that sometimes left participants clustered in a single data center. Configuration distribution bypassed Configerator, forfeiting canary rollouts and easy peer review. Kernel upgrades, access control, host decommissioning, and safe restarts all required reinvented wheels — duplicative code whose bugs affected only ZooKeeper, while Twine’s improvements passed it by.

Containerizing is the easy part

Running ZooKeeper on Twine meant satisfying both halves of the platform: the scheduler that places containers and manages their life cycles, and the containers themselves. The JVM-based ZooKeeper poses some resource-sizing challenges, but repackaging it was simple compared with safely integrating with the scheduler. The gold-standard mode is the Twine shared pool (TSP), a huge pool of managed hardware where platform engineers handle kernel updates, hardware remediation, and maintenance — but at the cost of strict constraints:

• CPU and RAM allocations are enforced by cgroups.
• The scheduler may evict containers at any time.
• Side effects on the machine are wiped once the scheduler decides to vacate it.
• Service owners are unprivileged guests.

That rule set suits stateless services. Teams needing disk persistence or tighter control can opt for private pools, but those come with the full burden of hardware ownership. Since the whole point of this migration was shedding fleet management duties, the team committed to making TSP work — despite the scheduler’s ruthlessness. Twine knows nothing about quorum requirements or data durability; it moves containers and deletes data as needed, even when those containers hold state that Twine itself depends on.

Durability on a stateless platform

A five-node ensemble needs at least three healthy participants at all times. Replacing a failed host means reconfiguring membership and syncing a new replica, which temporarily reduces replication capacity. Historically, those changes were done by hand, keeping intervention simple — but slow. Human reaction times meant a 12-hour response in a crisis, and the sheer number of hosts made maintenance a constant operational drain.

Twine, by contrast, replaced a failed host in under 90 seconds. That speed was impressive but also unsettling: Twine treats containers as opaque and does not copy filesystem data to replacements. The ZooKeeper layer must handle its own synchronization, while Twine marks the abandoned data set for immediate deletion. If the scheduler acted too aggressively during a botched replacement, the worst-case outcome was systematically destroying the infrastructure’s most valuable data.

The solution was to give the scheduler some quorum awareness. The team built one of the company’s first task controllers — a scheduler extension that requires Twine to ask permission before acting on ZooKeeper containers. For every operation, whether moving to a new host, starting or stopping a container, or resizing a job, the controller can acknowledge or reject it. The interface is simple but powerful: the controller enforces one-at-a-time operations, requires successful replication between them, and rejects anything that would endanger quorum durability.

By writing acknowledgements directly into the ensemble it protects, the controller persists state between operations and can detect when the ensemble is unreachable. This inverts the usual priority for Twine jobs: unhealthy ZooKeeper containers are rescheduled only when the rest of the ensemble can support the sync. Data preservation comes first, even when that means sacrificing service availability.

Breaking Circular Dependency Risks

Validating that ZooKeeper could run safely on Twine still left a critical concern: avoiding the circular dependencies that plagued our architecture diagrams. ZooKeeper supports numerous systems throughout the stack without creating risky dependency loops, so we first considered keeping core infrastructure ensembles on bare metal while moving only the rest. That approach would have forced us to maintain two distinct operational models, adding complexity without removing any. Instead, we committed to migrating all ensembles and engineered explicit escape hatches for every potential cycle in the dependency graph.

Although ZooKeeper serves data to many Twine-ecosystem systems, it does not power every facet of them. Schedulers cannot assign containers to hosts while ZooKeeper is down, but existing containers continue running. Service discovery cannot accept writes during an outage, yet established network endpoints still route traffic. We could therefore tolerate dependencies on those systems only where they were immune to ZooKeeper failures. This constraint produced four concrete capabilities:

• Starting and stopping containers without a functioning Twine scheduler.
• Injecting configuration into ensembles while the Configerator network could not serve reads.
• Informing ZooKeeper participants and select clients of an ensemble's new location while service discovery was rejecting writes.
• Obtaining packages and binaries while fbpkg's front end was offline.

Because ZooKeeper ensembles are few in number, recovery tooling could be tailored to a handful of cases. After restoring the fbpkg ensembles, for example, no subsequent recovery required the fbpkg escape hatch. These solutions were built using standard engineering practices:

• Container control: Each worker machine runs a daemon that manages containers per scheduler instructions. Twine team tooling lets an operator commandeer that agent and schedule tasks manually, using a JSON task specification that is trivial to back up and manipulate.
• Configuration: Since most ZooKeeper configuration is static and read at startup, build-time preparation can eliminate runtime dependency on configuration distribution. Dynamic configuration is written to the ensemble itself, like any customer data.
• Service discovery: Only a dozen ip:port tuples needed attention, not an entire fleet. Recovery tooling operates on raw IP addresses, and critical ensembles are assigned static, BGP-propagated virtual IPs that never change.
• Packages: New packages are initially stored in a low-dependency system before fan-out to the ZooKeeper-powered distribution network. With the fbpkg team, we built a recovery mode that serves packages directly from that source of truth when restoring critical ensembles.

These methods recover the ensembles that underpin low-level infrastructure during severe outages. But nothing prevented future dependency cycles from being introduced, and untested runbooks grow stale. To continuously validate our assumptions, we built a new testing framework that exercises the recovery runbook in a nearly empty environment where Facebook infrastructure is almost entirely unavailable. Every ZooKeeper release candidate and affiliated disaster-recovery tooling passes through this framework before being approved for production.

Correctness Under Load

Problems at scale often mean that a solution that works correctly anything less than 100 percent of the time will eventually fail catastrophically. A single ZooKeeper misstep threatened:

• Corruption of the source-of-truth data for all service-to-service RPC routing
• Unavailability of blob storage serving every image and video across our apps
• Deletion of millions of containers from our data centers

Two subtle failure modes illustrate the level of rigor required.

100,000 Membership Changes

Twine hosts reboot every few weeks for upgrades, and containers move continually for resource optimization. Each event requires replacing a ZooKeeper participant, leading to over 100,000 membership replacements over several years. The replacement process needed to be safe even if interrupted midway. Early designs that removed and added participants separately were rejected in favor of atomic swaps that actually healed misconfigurations on retry.

When a leader distributes a transaction, it counts followers that have recorded it to their write-ahead logs and issues a commit only once a majority have done so. Each participant's vote must be counted separately, so each is assigned a logical Server ID (SID). Twine assumes all containers in a job are fungible, yet ZooKeeper requires unique identities. The SID field is 64 bits, but uniqueness is needed on the lowest eight bits, too small a space for randomization. We initially used Twine's sequenced task identifier for each container in a job.

Task IDs alone proved insufficient. A network partition between one participant and the scheduler caused the scheduler to create a duplicate task on another healthy machine with the same identifier. After the partition healed, two participants briefly submitted durability votes on each other's behalf — a dangerous violation of consensus that could jeopardize data durability.

The fix required deriving a unique 8-bit identifier from both the task ID and task version number, which the scheduler assigns uniquely to each rescheduled participant. Combining these 40 bits of input into a unique 8-bit value seemed impossible until one engineer proposed using the current ensemble membership as a historical record of assigned identifiers. SID assignment logic XORs the full list of 64-bit participant identifiers, applies a bitmask, and yields the value in the 0–255 range that was least recently assigned. That value occupies the lower 8 bits of the next participant's SID.

Since ensembles never approach 256 participants, this incrementing assignment guarantees that no two contemporaneous tasks share a SID.

Kernel Behavior and Tail Latency

Some ensembles sustain heavy workloads, exceeding 100 million reads per minute and 500,000 writes per minute. Early in the Twine migration, these heaviest workloads occasionally exhibited latency spikes that matching bare-metal deployments did not. Investigation revealed a dozen violations of our one-second latency SLO every few hours, but only under specific load and hardware conditions.

Twine's container environment differs from bare metal in kernel version, TCP congestion algorithm, sysctl settings, and cgroups configuration. To isolate the cause, we systematically mutated container hosts to resemble their bare-metal counterparts, rolling back kernel settings and file system differences. The regression disappeared once we disabled the cgroup v2 memory controller, which enforces container memory limits. Disabling it effectively let the container consume the entire machine's RAM. The memory constraint was responsible for the occasional latency spikes.

Working with kernel maintainers and new container pressure metrics, we correlated the latency with elevated container-level memory pressure despite a healthy host. Using bpftrace to inspect kernel function timing, we traced each spike to a stall from direct page cache reclaim. Page reclaim frees filesystem cache pages when memory is scarce. Modern Linux performs this recycling asynchronously through the kswapd kernel thread, which activates only when the host experiences sufficient memory pressure. Containerizing ZooKeeper constrained the process's memory supply while other workloads reserved host RAM. Under high load, memory pressure triggered at the container scope but not globally, so kswapd never fired. The application paid the recycling cost inline with page requests, sometimes causing multisecond tail latencies.

Rather than awkwardly resizing containers, we went further: a kernel patch extends kswapd behavior to containers under pressure. Since its release, every container we run benefits, and the patch is being upstreamed to mainline Linux.

Elastic Operations

The full benefits of this migration emerged only after moving every ZooKeeper ensemble onto Twine. With ZooKeeper decoupled from physical hardware and manual placement, operations changed fundamentally:

• Fractional workloads: An ensemble now requires five containers, not five whole machines. Small workloads use fractional machines with strong isolation from neighbors and can be bin-packed efficiently on heterogeneous hardware.
• Dedicated ensembles: Sizing containers to workloads simplifies management of small workloads historically colocated in shared ensembles. Splitting them enables independent growth without fate sharing or noisy neighbors.
• Effortless scaling: Moving infrastructure needs to Twine pushed the marginal operational burden of each new ensemble toward zero, enabling deployments driven by customer demand rather than operational considerations.
• Novel automation: Twine APIs enabled fully automated ensemble life cycle management, unlocking workflow-driven live data migrations, ensemble splitting, regional turn-ups, and even short-lived ensembles for a single integration test.

Running the foundation on the scheduler

Making the lowest tier of infrastructure a proving ground for new operational models is uncommon in most organizations. With ZooKeeper running on the fully managed TSP, we've placed a stateful, low-level service atop the orchestration platform — a milestone that signals the platform is ready for our most fundamental components. It also offers a path forward for teams still managing their own hardware. The precedent is already spreading: other low-level, stateful projects like TAO and LogDevice are migrating onto the platform as well.

Operating a minimal-dependency service on a dynamic container orchestrator is a delicate exercise. Migrating hundreds of ensembles without disruption demands persistence, and running the orchestration platform itself on the very containers it manages forces a hard look at disaster recovery — and a measure of audacity. In return, ZooKeeper's infrastructure is now positioned to move quickly while remaining dependable, and the lessons learned here will inform how other foundational services approach the same transition.