Measuring Kafka’s ceiling under production conditions

Dropbox runs Kafka across several critical systems, including analytics, machine learning, monitoring, search, and its Cape stream-processing framework. The Jetstream team, which manages these clusters, needed a reliable way to determine how much traffic a broker can handle before it degrades. That number drives provisioning decisions, so the team built an automated test platform to map out Kafka’s throughput boundary in their infrastructure.

The test setup uses Spark to host Kafka clients, making it straightforward to scale producer and consumer traffic. Three Kafka clusters of different sizes were stood up, with a dedicated test topic used to generate load. To keep writes evenly distributed, the topic uses 10 times as many partitions as there are brokers, so each broker leads exactly 10 partitions. That ratio avoids write contention from too few partitions per broker, which the team found to be a throughput limiter below that threshold.

Because Dropbox infrastructure spans multiple U.S. regions, test clients are distributed as well. Since traffic stayed well below the capacity of the network backbone, the team assumes the limits measured over cross-region links also hold for local traffic.

Narrowing down the variables

Kafka workload characteristics are shaped by many parameters: producer count, consumer group count, initial consumer offsets, message rate, message size, and topic/partition layout. Testing every combination would be impractical, so the team searched for the dominant factors. Their conclusion: the two components of raw throughput—messages per second and bytes per message—account for most of the variance.

That result lets them model the problem as a two-dimensional traffic space. Each traffic pattern is a point on a plane defined by messages per second and bytes per message. The set of patterns a cluster can handle forms a closed region, and the cluster’s throughput limit is simply the border of that region.

Automating boundary discovery

Plotting that boundary requires hundreds of test runs, which is not something that can be done by hand. The team designed an automated algorithm to probe the limit, but it first needed a reliable, programmatic way to detect overload.

Two indicators, both already used for health monitoring in production, were selected:

  • IO thread idle time below 20%, signaling the request-handling thread pool is saturated.
  • In-sync replica set changes more than 50% of the time, indicating at least one broker can’t keep up with replication.

To find a single boundary point, the team fixed the bytes-per-message value and varied load until Kafka started failing. Rather than tuning the per-producer message rate directly—which is hard to control due to batching—they adjusted the number of active producers, each running at the same produce rate. Producer count scales traffic linearly, and earlier experiments showed that increasing producer count alone doesn’t meaningfully change Kafka’s load profile.

The search itself is a binary search. It starts with a wide window of producer counts from zero to a value known to overload the cluster. Each iteration tests the midpoint; if Kafka is overloaded, that value becomes the new upper bound, otherwise it becomes the lower bound. The process repeats until the window narrows to an acceptable margin, and the message rate at the lower bound is recorded as a boundary point. Repeating this across a range of message sizes produces the full boundary curve.

What the boundary looks like

The plotted boundaries for clusters of different sizes all converge on the same per-broker ceiling: 60 MB/s. This is a conservative figure, since test messages were fully randomized to minimize the effect of Kafka’s internal compression. At the limit, both disk and network are heavily utilized. Real-world messages often follow repeating patterns, leaving room for compression. In an extreme test where all messages contained the same character, throughput climbed well past 60 MB/s as disk and network pressure eased.

The limit holds with up to five consumer groups reading from the test topic, meaning the write throughput is sustainable when read traffic is five times larger. Past that ratio, reads start to consume enough network bandwidth that write throughput declines. Since production read-to-write ratios at Dropbox stay well below five, the 60 MB/s figure applies to all production clusters.

The result also informs how much headroom to budget. If a cluster should tolerate 20% of brokers going offline, the safe per-broker throughput is about 60 MB/s times 0.8, or roughly 50 MB/s. That gives a simple formula for sizing future clusters from projected throughput.

Future uses

The automated platform isn’t a one-off measurement tool. The same test suite can be rerun whenever the environment changes—new hardware, different network configuration, or a Kafka upgrade—to quickly establish a new throughput baseline. The methodology can be extended to other performance factors beyond message rate and size. And the platform doubles as an isolated test bench for simulating new traffic patterns or reproducing production issues.

One caveat: these numbers are specific to Dropbox’s hardware, software stack, and network topology. Other operators will see different results, but the technique itself—defining a traffic space, detecting overload via health metrics, and automating a binary search for the boundary—can be applied to understand any Kafka deployment.