What Was Wrong With the Original

Dropbox runs scribe as its log aggregation backbone, feeding an analytics pipeline from a three-tier fan-in tree: leaf nodes on every machine, a remote hub buffering tier, and central hubs that write to HDFS. A recent outage prompted a full rewrite. The team’s goals were straightforward: cut operational overhead, reduce data loss, and add capabilities the original scribe simply lacked. Here is how they approached it.

Configuration Without Restarts

The original scribe loads an XML-like config file at startup. Any change—say, adding a remote hub node—requires editing files across thousands of leaf nodes and restarting the entire fleet. That process does not scale to tens of thousands of nodes, and it is nearly impossible to guarantee every node runs the latest config.

The rewrite stores all configuration in ZooKeeper. Nodes pick up changes and reconfigure themselves automatically, with no restart required. Service discovery is built in, so upstream scribe nodes can change without manually updating every downstream config.

Throttle on the Way Out, Not In

Old scribe throttled at ingress. When a batch of messages arrived, it checked the queue for each category in the batch; if any queue was full, the entire batch was rejected and the client told to retry. For multi-category batches, this causes head-of-line blocking—one busy category stalls the rest. Worse, a client that ignores the “try again” response loses messages permanently, and retries burn bandwidth and CPU.

New scribe always accepts incoming messages and throttles on egress. Messages are buffered until the category queue is full. Each upstream writer is rate-limited with a leaky bucket, which prevents upstreams from being overwhelmed without dropping data at the front door.

Memory and Disk as Separate Queues

The original scribe treated each category’s memory buffer and disk buffer as a single logical queue—serializing the two and forcing disk writes even when memory has capacity. The rewrite separates them. Messages go to the memory queue first; only when that is full does the disk queue get used. The upstream writer pulls from both queues simultaneously, so the memory queue keeps flowing while the disk queue catches up. This reduces disk I/O pressure. Message ordering is not preserved, but that is acceptable because the analytics pipeline already has to handle out-of-order messages.

Sharding and the Single Point of Failure

Central hubs each own specific categories because HDFS does not allow multiple writers to append to the same file. That makes every central hub a single point of failure and a throughput bottleneck. The application client deals with this by sharding high-throughput categories, appending a shard suffix, and merging downstream. Routing is predetermined, though, so if a central hub dies, its messages just wait until the node is replaced.

For sharded categories, the rewrite ignores the client-provided shard suffix. Instead, it assigns a new shard suffix round-robin style each time it forwards a batch upstream. That rotates which upstream gets the traffic and keeps messages moving past a dead node. Unsharded categories still have a single point of failure, but forcing every category to have at least two shards solves it.

Disk Buffering That Survives Corruption

Old scribe wrote a new file for every disk flush, all into one directory—a recipe for OS trouble at scale. Files were un-checksummed, and on corruption the node could enter an endless crash loop. Loading a file for upstream forwarding meant reading it whole and sending the entire contents as one message batch.

The rewrite uses checksummed logs with rotations and checkpoints. A separate reader thread handles immutable log files while a writer thread appends to a single mutable file. Rotation to immutable happens when the reader is starving or the file hits a size threshold. Each category gets its own subdirectory on disk.

Kafka Support Out of the Box

The analytics team wanted to move from HDFS to Kafka for access to tools like Storm. Old scribe would have required a scribe-to-Kafka shim or a service tailing HDFS files—another server in an already complex stack. The rewrite natively supports Kafka as an upstream destination, which removes that extra hop entirely.

Two Architectures, One Goal

To make the design differences concrete, it helps to compare the block diagrams side by side. OldScribe and NewScribe are both structured like a network switch, but they differ in one critical way: where routing and configuration logic lives.

OldScribe: Static Routing

OldScribe runs each category in its own thread. The individual components within a category are just library calls into a common abstract Store interface. There is no control plane here; routing configuration is loaded once during startup and never updated. You can think of it as a switch with a fixed forwarding table.

NewScribe: A Split Control and Data Plane

NewScribe keeps the same conceptual shape but introduces a proper control plane that runs alongside the data path, enabling live reconfiguration without restarts.

The Control Plane

  • Config manager: Watches ZooKeeper for any configuration changes. When a change occurs, it notifies the category config updater.
  • Upstream manager: Owns the connection pools for Scribe and Kafka upstreams. It tracks discoverable Scribe upstream services and alerts the category config updater whenever the upstream set changes.
  • Category config updater: Receives notifications from both the config manager and the upstream manager, and propagates updates to every registered category's throttling, buffering, and upstream settings.

The Data Plane

  • Memory queue: A standard thread-safe in-memory queue.
  • Disk queue: A background writer thread appends messages to log files while a background reader thread actively loads messages back from those files.
  • Upstream writer: Pulls events from both the memory and disk queues and sends them to the upstreams configured by the control plane.

The rewrite is tightly coupled to Dropbox's internal infrastructure, so the code cannot be open-sourced at this time. Still, the architectural split described here should give you enough to re-implement the approach on your own stack.