Why Your Latency Dashboards May Be Lying to You

Most service owners at Twitter rely on latency metrics for dashboards and alerting. The numbers they typically use, however, often measure something different from what they actually want to track. Three historical quirks in the measurement setup distort the picture: latency that accumulates in uninstrumented parts of the stack, the inability to aggregate metrics across a cluster, and a coarse minutely sampling resolution.

Opaque Latency Hides Where Time Is Lost

The pipeline a client request traverses is long: client netty, client Linux, the network, server Linux, server netty, and only then server user code, before the response makes the same journey in reverse. Server-side metrics only capture time spent in server user code. Client-side metrics capture everything except client user code, including queueing in netty and the Linux kernel on both ends.

This opacity matters because queueing in netty and below is common, especially under high load. Kernel CPU throttling from CFS bandwidth control can add significant invisible latency precisely when dashboards are most needed. When alerting on client latency, a service owner can be paged for queueing on the client machine while the service itself is running perfectly fine.

Client metrics also aggregate latency across all servers a client talks to, while server metrics are per-instance. There is no clean way to reconcile these views or to determine, from metrics alone, whether opaque latency originates on the client or server side.

The gap can be large. Cluster-wide CDFs comparing client- and server-measured latency show a 99th-percentile latency of roughly 16ms at the server versus 240ms at the client, a 15-fold difference. A latency that is at the 99th percentile client-side is well above the 99.9th percentile server-side even on log-scaled axes where the lines appear close. The delta is often small at lower percentiles but widens at the tail, where load drives queueing in the uninstrumented layers.

A scatterplot of client versus server latency for individual requests makes the problem concrete: for a request with 10ms server-measured latency, client-measured latency can be as high as 500ms. Most requests show similar client and server times, but a meaningful subset has server metrics that are wildly inaccurate representations of the client experience. Instrumentation through netty and the kernel is possible but not imminent. To estimate opaque latency today, Zipkin tracing data sampled uniformly at random provides a workable approximation.

Aggregating Shard Metrics Distorts Tail Latency

Most services rely on metrics that precompute fixed aggregations per shard. These cannot be combined into a valid cluster-wide figure. The most common dashboard target is a per-shard average of shard-level 99th-percentile latency, but averaging tail latencies defeats the purpose of monitoring tails. With high fanout and deep request trees, a tiny fraction of slow server responses can slow down most top-level requests. An average of per-shard tails completely misses this effect. It also fails to capture the benefits of a true cluster-wide average, which is reconstructible from per-shard averages when tail metrics are not involved.

Consider a 100-node cluster where tail latency rises 10x on one node. The average of per-shard tails increases by roughly 9% even though the cluster-wide tail is dramatically elevated. Host-level issues that drive tail latency on a single node up by an order of magnitude are frequent enough that this is not a hypothetical concern.

Some dashboards attempt a percentile of per-shard 99th-percentile values, often the 90th or 99th. This does not work either; no per-shard aggregation can reconstruct the cluster-level tail. Plots of various dashboard aggregations against measured cluster-wide 999th-percentile latency show weak correlation. Even the aggregation with the best average behavior is materially wrong for most individual minutes sampled. The only real fixes are extending tracing pipelines for dashboard and alert use or adding metric histograms to Finagle and plumbing them through the metrics stack. Popularity of the average-of-tails approach does not make it sound; it has neither the properties owners expect nor the properties they want.

Minutely Metrics Miss Bursty Failures

Metrics collection at one-minute granularity is an independent blind spot. Sub-minutely events can be the root cause of incidents while remaining invisible in standard dashboards.

One incident provides a clear example. A service showed elevated latency and error rates, but standard metrics were uninformative. Sub-minutely data immediately revealed the mechanism: a large latency spike at time zero, followed by 30 seconds of near-zero request rate to a particular cache shard. The requesting service was configured to mark servers as dead for 30 seconds after enough failed requests, and the decision was distributed, so only some shards stopped sending traffic while others continued. The reported p99 latency for the cache instance was 0.37ms; the actual sampled client-observed p99 was about 580ms, a gap of over three orders of magnitude, largely in invisible parts of the stack.

Sub-minutely visibility alone would not have caught this. The elevated latency was concentrated in uninstrumented layers, so monitoring cache latency metrics was insufficient regardless of resolution.

Summary

The existing setup mostly works; reliability is comparable to or better than peer companies of similar scale. The costs are real, though. Incidents frequently require specialized tracing tools that most engineers do not use routinely, which adds on-call toil. And large errors in cluster-wide latency estimates force conservative provisioning and latency SLOs much stricter than actual targets to prevent user-visible problems, raising operating costs.

Appendix: Open Versus Closed Loop Benchmarks

Synthetic benchmark setups sometimes use closed-loop measurement, issuing a request, waiting for completion, then issuing another. Some permit N requests in flight but retain similar realism problems.

In a toy example, a service receives exactly one request per second with a normal response time of half a second. Under an open-loop benchmark issuing requests at one per second, all percentiles sit at half a second. If one request in the middle of a one-minute run takes ten seconds and the service has no parallelism, an open-loop setup will observe the queued requests behind it. A closed-loop setup that waits for each request to finish before sending the next will hide that queueing entirely. Computations of mean and 90th-percentile latency differ substantially between the two. For a deeper treatment, see Nitsan Wakart's write-up on coordinated omission in YCSB or Gil Tene's presentation on the topic.

Appendix: Unweighted Averages Mislead

An independent and common dashboard error is the unweighted average, frequently applied across datacenters or across shards to produce an overall latency figure. Shards under lower load tend to have lower latency. When traffic fails away from a datacenter, an unweighted average across datacenters often shows decreased latency even though the requests actually served experienced increased latency. This compounds the other measurement issues but is avoidable with weighted aggregation.