Netflix Opens Up Its ZooKeeper Toolkit

Netflix has long been a proponent of open source software, and today it formalized that commitment with a new portal on Github hosting its first wave of projects. The initial lineup includes Curator—a library focused on Apache ZooKeeper—along with Astyanax (a Cassandra client), Priam (a co-process for Cassandra backup and configuration), and CassJMeter (a JMeter plugin for Cassandra testing).

The Pain Points of ZooKeeper

ZooKeeper is a high-performance coordination service that provides naming, configuration management, synchronization, and group services for distributed applications. While it ships with a Java client, that client places a significant amount of manual housekeeping burden on developers.

Several classes of problems make the built-in client difficult to use correctly:

  • Connection management. The client performs a handshake with the server at startup, and any synchronous method calls such as create() or getData() will throw exceptions until that handshake completes. If the connection is lost, failover to another cluster node resets the client to this initial connection state. Session expiration can also occur in edge cases, requiring users to manually close and recreate the ZooKeeper instance.
  • Recoverable errors. When creating a sequential ZNode, the server can successfully create the node but crash before returning the node name to the client. The client also throws several recoverable exceptions that users are expected to catch and retry.
  • Recipe complexity. The standard ZooKeeper "recipes"—locks, leader election, and similar patterns—are described only minimally in the documentation and are notoriously subtle to implement correctly. Important edge cases, such as the lock recipe's failure mode when a server crashes mid-creation, are not covered and can lead to deadlocks. Leader election, for instance, must account for connection instability: a leader cannot safely assume it retains leadership if the connected server crashes, until failover to another server succeeds.

Every ZooKeeper user must address these issues, and the solutions are neither trivial nor obvious. Curator aims to handle all of them.

What Curator Provides

Curator (named for a "keeper or custodian of a collection") is structured as three related projects that sit on top of ZooKeeper's native API:

  • curator-client — An alternative to the bundled ZooKeeper class, handling low-level connection housekeeping and offering useful utilities.
  • curator-framework — A high-level API that simplifies ZooKeeper usage by managing cluster connections and automatically retrying operations.
  • curator-recipes — Implementations of common ZooKeeper recipes, built atop the Curator Framework.

The primary focus is on the recipes layer. Most users of ZooKeeper don't want to manage connection internals—they want a reliable, simple way to use locks and leader election. Curator is designed around that goal, reducing ZooKeeper complexity through several key mechanisms:

  • Pluggable retry mechanism. All operations that generate recoverable errors are retried per a configured policy. Several standard policies, including exponential backoff, are bundled with the library.
  • Connection state monitoring. Curator continuously watches the connection state and exposes listener hooks for users to respond to changes.
  • Instance management. While Curator uses the standard ZooKeeper class for the actual connection, it manages the instance internally and recreates it as necessary, providing a more reliable handle to the cluster.
  • Correct recipe implementations. The bundled recipes are written with ZooKeeper best practices and account for the known edge cases described above, including the crash-during-creation problem that commonly causes deadlocks in hand-rolled code.

ZooKeeper in Production at Netflix

Curator is already in extensive use across Netflix's infrastructure. Notably, the company relies on it for Cassandra backups, its TrackID service, and various caching layers. The InterProcessMutex recipe secures unique values in sequence ID generators, the LeaderSelector handles housekeeping tasks in its Chukwa collector, and the InterProcessSemaphore manages concurrency limits when interacting with third-party services that cap concurrent users. That last example is a good illustration of how the recipes shift concern away from connection plumbing and toward actual application logic—the team could rely on ZooKeeper's coordination features without writing that failover and retry handling themselves.

Getting the Code

Curator's binaries are available from Maven Central, making the library straightforward to add as a dependency. The full source code is hosted at https://github.com/Netflix/curator, with extensive documentation available at https://github.com/Netflix/curator/wiki.