Why Tracing Feels Unuseful—and What Actually Fixes It

There’s a common refrain in engineering circles: distributed tracing is hard to justify, especially outside the scale of a Facebook or Google. The doubt is understandable. Early tracing systems often delivered so little value for their cost that they got abandoned. But most of the pain traces back to a handful of concrete infrastructure flaws, not to tracing itself being a dead end.

A trace-view-oriented approach—where you store raw spans and provide a UI for viewing individual traces—fails in predictable ways. The schema makes queries expensive, aggregation is practically impossible, sampling is wildly non-representative, and clock skew destroys any notion of reliable timing. Once those root causes are fixed, tracing starts paying for itself.

The Four Obstacles That Kill ROI

Schema That Fights Queries

In the typical system, a trace is a set of spans, each span is a set of annotations, and every span points to its parent so the graph can be re-built. That structure is flexible but expensive to ask questions of. Any query touching the graph structure requires reconstructing the whole trace from every span. Without per-trace summaries, even simple queries have to read every span. In practice, the only viable query becomes “show me a few spans matching this filter.” Anything richer times out.

No Aggregation, Only Navigation

Without aggregation, the trace UI becomes a narrow porthole into recent raw traces with a service-name filter. You can’t ask whether the trace you’re looking at is representative, isolate tail events like a timeout between service A and B, or even filter for complete traces. Especially painful: a trace that’s “too large” simply fails to render, a direct consequence of combining the schema’s limitations with no rate limiting.

Unknowable, Non-Representative Sampling

Sampling decisions were scattered across code with no central documentation. Compounding that, spans had multiple drop points: the local agent, a collector that regularly lost nodes, and a backing database that rejected writes under load. Since the trace id was also the database key, a large fan-out request produced a burst of writes to a single key—exactly the kind of load hot spots cause.

The more subtle sampling flaw: an independent sampling decision at every RPC. Most traffic consists of leaf calls, so a moderate sampling rate yields mostly single-span traces hanging off the service graph’s edge. A single root-span trace, where sampling only occurs if a span has no parent, produces complete traces with uniform probability. That fix alone eliminates an enormous amount of wasted storage and bias.

Clock Skew

Timestamps simply don’t align across hosts. Even within one host, naive duration computation produced negative spans. The existing pairwise adjustment logic didn’t repair the problem; it just made the accounting messier.

The Fix: Make the Data Queryable

One set of fixes is straightforwardly operational. Drop-prone collector nodes went away in favor of a real queue that absorbs bursts and paces writes. A two-minute garbage-collection tuning reduced collector crashes by a factor of about a hundred as an interim step.

The larger mindset shift—building datasets and tools that answer questions, rather than just storing raw traces—comes straight out of the Dapper paper and takes most of the value. The enabling technical change is a set of ready-made tables that store precomputed, query-friendly views of trace data:

  • trace_index—trace-level facts like root information and request endpoint.
  • span_index—client and server side details per span.
  • anno_index—standard annotations like payload sizes and send/recv timestamps.
  • span_metrics—precomputed metrics like span duration.
  • flat_annotation—all annotations, for ad hoc queries.
  • trace_graph—the structure itself, for graph queries.

These tables, partitioned daily and queryable with SQL (or Scalding/Spark for heavy graph jobs), turn tracing into something people actually use. The questions they unblock include:

  • A representative set of traces for a struggling service.
  • Which upstream service causes elevated load.
  • Where unusual write amplification appears, including whether it is specific to a call path rather than universal for a service.
  • Time spent on serialization versus actual work.
  • Cost of different kinds of requests in backend effort.
  • The latency critical path for a high-latency mobile request.
  • Which downstream services a given endpoint actually touches, keeping in mind call patterns are data-dependent.

Going to the UI

Queries lower the entry barrier, but a better trace view can eliminate the need to write SQL at all. Trace data is richer than metric data, which paradoxically makes raw trace views harder to get value from. A dashboard of graphs is intuitively useful; a raw list of traces is not.

The Service Dependency Explorer (SDE) is a concrete step past raw views. It replaces the “sea of nodes” visual, which becomes unreadable with thousands of services, with a higher-level, focused summary. Given a service, it shows direct callers and callees, load amplification ratios, and what share of traffic follows which edge. In a controlled example, it clearly shows 20x amplification from E to C, and 4x to indirect callee D. In a real-world case, it can summarize a particular day with:

  • 10 total front-end-rooted traces
  • 110 total traced RPCs to a given service
  • 2,100 total traced RPCs caused by that service
  • 3 unique call paths from front-end endpoints

From there, a click reveals that a specific call downstream never hits three services that another call from the same service does—or that upstream dependencies differ by endpoint, not just by service. That level of exploration surfaces unusual load amplification, the causes of load, and cycles in the call graph, all without running a query.

Complementary, Not Either/Or

Tracing and metrics are not substitutes. Some problems only appear in trace data—for example, discovering that a specific call path, a few hops away from the point that exhibits unusual load amplification, is responsible. In metrics-only analysis, that relationship gets buried.

The full power emerges when they are joined. Consider the impact of a bad host on latency. You don’t annotate spans of a bad host at runtime, because if you knew it was bad, you wouldn’t be running it. But historical host data and traced critical paths can be combined to measure end-to-end impact after the fact.

The infrastructure work described here was incremental. Rough analyses with existing tools and staffing eventually produced the datasets. It is not a story about genius engineering or exotic tooling—less than two person-years went into everything discussed. The difficult parts were the same and intractable problems any project faces: getting organizational support and maintaining a clear-eyed view of what question you’re trying to answer.