Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned

From Prototype to Production: The Hard Part

The first version of our service topology system worked exactly as designed—in a local environment. In production, it quickly became clear that the gap between a working prototype and a system that can operate at Netflix scale is vast. Kafka consumers fell behind. Instances exhausted memory. A small number of nodes received 100x the traffic of others, and garbage collection pauses consumed more CPU than the actual business logic.

This post covers the engineering reality of scaling that system: the architectural decisions that made scale possible, the production failures that exposed their weaknesses, and the methodology we used to iterate toward a solution. The goal was ambitious—process millions of flow records per second, reconstruct service topology at any point in time, answer queries in under a second, and keep everything fresh in near real-time.

Streaming Architecture for High-Throughput Ingest

The ingestion layer is the first place where theoretical design meets practical throughput limits. We rely on multiple data sources—eBPF network flows, IPC metrics, and distributed tracing—each arriving as a high-volume stream that must be consumed, normalized, and aggregated without falling behind.

Early on, we discovered that a straightforward Kafka consumer setup was insufficient. A single consumer group processing all partitions led to uneven load distribution and frequent lag. The fix came in two parts. First, we partitioned the input streams by service pair, which allowed independent consumers to process disjoint subsets of the data in parallel. Second, we moved the heavy lifting—aggregation and graph construction—out of the streaming layer entirely and into a separate pipeline that could be scaled independently of intake.

This decoupling proved critical. The streaming consumers became lightweight: they only needed to parse records, extract the service pair key, and push events to the next stage. All stateful computation, including windowed aggregation and edge weight updates, moved to a distributed processing tier that could be scaled horizontally based on the observed lag rather than on traffic volatility.

Distributed Aggregation with Hot-Key Handling

Aggregation is where the system’s scale challenges truly surfaced. The core operation—counting flows between service pairs over a sliding window—sounds simple, but real traffic is not uniform. A handful of high-traffic service pairs dominate the stream, creating hot keys that single nodes cannot handle.

We initially keyed all aggregation state by the service pair. This made lookups trivial but caused severe skew: the node responsible for a hot pair would saturate while others sat idle. The solution was to introduce a two-level aggregation design. The first level runs on the streaming consumers that ingest the data, performing a partial aggregation over short time windows. This reduces the data volume before it reaches the global aggregation stage, which then merges partial results.

The two-level approach also gave us a natural mechanism for handling late or out-of-order events. The partial results carry timestamps from the ingestion stage, allowing the global stage to apply watermarks and drop events that fall outside the allowed delay. This prevented a single delayed batch from causing a cascade of reprocessing across the pipeline.

The Storage Layer: Graph State and Query Performance

The aggregated topology needs to be stored in a way that supports both point-in-time reconstruction and fast traversal queries. We evaluated a range of options—from relational databases to specialized graph stores—and settled on a design that separates the concern of current state from historical state.

For the current topology, we use an in-memory graph that lives alongside the aggregation tier. This graph is updated incrementally as new partial results arrive and is snapshotted to durable storage at regular intervals. The snapshot serves two purposes: it provides a recovery point for restarts, and it acts as the baseline for historical queries when combined with a change log.

Historical queries, which require reconstructing what the topology looked like at a specific time, are handled by a time-travel mechanism. Instead of storing every version of the graph, we store the initial state and a compact sequence of deltas—each delta representing the changes that occurred within a small time window. Reconstructing a past state simply involves applying the deltas up to the desired timestamp, which we found to be significantly faster than replaying raw events.

Sub-Second Query Responses

Query latency is a product of both the storage format and the query engine. For the interactive use cases—finding dependencies of a service, computing blast radius, or exploring a merge of layers—we needed responses in the hundreds of milliseconds, not seconds.

The in-memory graph is stored in an adjacency list format optimized for traversal. Each node points to its outbound and inbound edges, and edge properties such as protocol, volume, and recent activity are embedded directly in the graph structure. This avoids the overhead of joins or lookups during a query. We also precompute certain derived views, such as the set of services that depend on a given service either directly or transitively up to a fixed depth.

For merging multiple graph layers—for example, combining IPC metrics with tracing data—we perform the merge at query time by reading from the relevant layer stores and combining the results in memory. This keeps the write path simple and avoids the consistency issues that would come from maintaining a denormalized combined graph. The trade-off is that complex merged queries can be slower, but the precomputed views cover the common cases well enough to meet the latency budget.

Operational Lessons

The production rollout taught us lessons that apply well beyond this specific system.

First, load testing with realistic data is non-negotiable. The local environment did not expose memory pressure or garbage collection issues because the data volumes were orders of magnitude smaller. We had to build a synthetic traffic generator that could replay production-like patterns, including the hot keys and the spikes, to find the real bottlenecks.

Second, monitor the consumers before you monitor the graph. The health of the entire pipeline is determined by how well the streaming layer keeps up. We instrumented consumer lag, processing time per record, and memory usage per partition, and we set up alerts that would page us before the backlog became unmanageable.

Third, design for skew from the start. Even if your traffic is balanced today, it will not be tomorrow. The two-level aggregation and the partitioning strategy are what made the system resilient to the uneven traffic patterns that we now see every day.

Conclusion

Building a real-time service topology system at scale required more than just a clever algorithm—it demanded rethinking how we ingest, aggregate, store, and query distributed data. The architecture we settled on—streaming intake, two-level aggregation, in-memory current state with delta-based history—emerged from a series of production failures and the methodology we used to address them. The result is a system that can process millions of flow records per second, answer time-travel queries, and do it all without falling behind. The path to get there was not linear, but each challenge forced a design that was more robust than the one before it.

The Architecture: Streaming Over Batch

Topology systems built on batch processing share a common failure mode: they answer yesterday’s questions. When an incident happens at 3am and the dependency map is an hour old, that map is closer to archaeology than observability. The design decision that shaped this system was to go streaming-first from the outset, ingesting flow records from multi-region Kafka and IPC metrics over Server-Sent Events, then pushing updates through reactive pipelines to deliver topology changes within tens of minutes rather than hours or days.

Real-time delivery isn’t a luxury here. Live event monitoring can’t wait for a scheduled batch, incident response needs current state, and change validation only works if the impact is visible immediately. The architecture had to process millions of flow records per second continuously without falling behind.

Backpressure as the Enabler

Continuous ingestion at that volume creates a hard question: what happens when downstream systems slow down? The common answers all break at scale. Unbounded queues buffer until memory runs out and the instance dies. Drop-based flow control keeps things fast but corrupts the topology by losing connection data. Batch processing avoids both problems by making the data stale enough to be useless.

Reactive streams with backpressure solve this differently. The pipeline slows down rather than losing data. When the final stage can’t write to the graph database quickly enough, it signals the previous stage, which signals the one before it, all the way back to the Kafka consumer, which pauses and lets data accumulate in Kafka until capacity returns.

This propagation mechanism means the entire system self-regulates. Traffic spikes, GC pauses, or external slowdowns at any stage automatically cause a graceful slowdown to a sustainable rate. Data isn’t dropped in most cases, instances stay alive, and the pipeline degrades smoothly. Normal operation runs at minimal latency; under load, updates arrive seconds or minutes late instead of being lost or delayed for hours. For a topology view, that trade-off is entirely acceptable.

The cost is real complexity. Reactive streams are harder to reason about than blocking synchronous code, but backpressure isn’t a design nicety here; it’s the mechanism preventing catastrophic failure under production load.

Physically Separated Layers

Three topology layers exist independently, each with storage tuned for its data type:

• Network Layer: eBPF flow logs persisted in a graph database partition, offering full coverage but no application context.
• IPC Layer: application metrics in a separate graph database, rich in endpoint detail but only covering instrumented services.
• Tracing Layer: distributed traces in columnar storage (Parquet), reflecting actual request paths but sampled.

Physical storage isolation buys independent optimization: each layer has different throughput patterns, query loads, and evolution timelines. Queries run in parallel across the relevant storage systems and merge at the edge, presenting a unified view with sub-second latency while allowing each layer to change on its own schedule.

The Three-Stage Pipeline for Network Flows

Network flow logs have an inherent problem: they capture individual hops, not application-level connections. Traffic between apps in a cloud environment rarely travels directly. Load balancers, NAT gateways, API gateways, and proxies all sit in the path. Flow logs show App A → Load Balancer and Load Balancer → App B as separate flows, leaving engineers to decipher infrastructure noise instead of seeing the logical dependencies that matter for troubleshooting. Resolving those intermediaries into clean App A → App B edges is the core problem the pipeline solves.

Stage 1: Initial Aggregation

Multi-Region Kafka (4 regions)
  → Filter invalid flow logs
  → 5-minute time-window batching
  → Create initial aggregators per window
  → Distribute via consistent hashing
  → Stream to Stage 2 via SSE

Stage 1 consumes flows from multi-region Kafka, filters invalid records, and batches them into 5-minute windows with initial aggregator objects. These are raw network hops, classified by whether they involve intermediaries but not yet resolved. Aggregators then stream to Stage 2.

Stage 2: Intermediary Resolution

Stage 1 Aggregators (via SSE streams)
  → Group flows by intermediary (load balancer, NAT gateway, proxy, etc.)
  → Identify pairs: (Source → Intermediary) + (Intermediary → Destination)
  → Resolve to direct edges: Source → Destination
  → Track which intermediaries were traversed
  → Aggregate metrics across both hops
  → Re-distribute via consistent hashing
  → Stream to Stage 3 via SSE

Stage 2 does the graph resolution work. It groups aggregators by intermediary, building maps of incoming flows (Source → Intermediary) and outgoing flows (Intermediary → Destination). For each intermediary, it joins those sets to create direct application edges, combining metrics from both hops. The result is clean application-level topology: App A → App B, not App A → Load Balancer → App B.

This join happens at aggregation time, not at query time. The reason it can’t happen in Stage 1 is data locality. Joining App A → Load Balancer with Load Balancer → App B requires both flows to be colocated on one instance. Kafka’s partitioning in Stage 1 scatters related flows arbitrarily across instances. Stage 2 exists to redistribute aggregators by intermediary identifier, so every flow touching “Load Balancer X” lands on the same instance, the classic map-reduce pattern: Stage 1 maps, Stage 2 shuffles and reduces, Stage 3 finishes the aggregation.

Stage 3: Enrichment and Persistence

Stage 2 Aggregators (via SSE streams)Flow
  → Final aggregation across time windows
  → Enrich with external data (query key-value stores)
  → Convert to graph entities
  → Persist to graph database (throttled writes)

Stage 3 performs the final aggregation of resolved edges, enriches graph nodes with external data like application health, ownership, and metadata, then converts aggregators into concrete graph entities with all properties populated. Writes go to the distributed graph database with controlled throttling to respect storage limits.

Why Three Stages Instead of Two

The initial design had two stages: aggregate in Stage 1, then resolve and persist in Stage 2. That worked in testing and fell apart in production. Intermediary resolution requires collecting all flows for a given intermediary on one instance, which means the instances handling popular applications became hot nodes with severe data concentration. On top of that, the enrichment step, querying external stores for health and metadata, piled the heaviest I/O onto the busiest instances.

Splitting into three stages isolates the concerns. Stage 2 purely resolves, then redistributes. Stage 3 handles enrichment and persistence. Rather than pointing all flows for a hot key at one owner, each flow is distributed, resolved, distributed again, and then persisted. Work spreads across instances, and compute-heavy resolution stays separated from I/O-heavy enrichment. Even intermediaries with 100x normal traffic don’t create a single-instance bottleneck.

Why SSE Rather Than gRPC or Message Queues

gRPC was the initial transport between stages, but it became a performance bottleneck. Serialization overhead, connection pool management, and memory pressure for streaming responses consumed more CPU than business logic. Message queues added infrastructure complexity with no payoff for this use case.

Server-Sent Events turned out to be the right tool: lightweight HTTP with minimal serialization, natural backpressure integration with reactive streams, and a much simpler connection model. The lesson is that even sound industry defaults like “use gRPC for service communication” need verification against the actual workload. For streaming large volumes of pre-aggregated data, the lighter-weight option won. Measure, don’t assume.

Why IPC Skips the Pipeline

The IPC layer aggregates in a single stage for two reasons. First, IPC metrics are already at the application level, so there are no intermediaries to resolve. Second, the data arrives with correct partitioning: consistent hashing assigns each node all IPC metrics for its applications. No redistribution needed. The architecture principle here is simple: the data partitioning strategy dictates the processing architecture. Properly partitioned input aggregates directly; improperly partitioned input, like network flows requiring intermediary resolution, needs shuffle stages in between.

Dynamic Hashing With Autoscaling

Auto Scaling Groups that add and remove instances on demand would normally break partitioning schemes that assume a static cluster, forcing explicit rebalancing, coordination services, or manual data movement. None of that happens here. Each instance queries the service registry for the current list of healthy ASG instances, keeps them sorted (so all instances share the same view), and uses that list with the hash function findOwnerInstance(aggregator.primaryKey) to decide ownership.

When the ASG scales, the updated instance list changes the hash function’s output and aggregators redistribute automatically. No coordination protocol required. The registry already tracks ASG membership for health checking, so dynamic cluster membership comes for free. Consistent hashing keeps most aggregators stable on their current instances during membership changes while the sorted list guarantees consistency across all nodes.

Load follows infrastructure as a side effect. Spikes and live events trigger scaling that puts new instances to work immediately. Deployments shift aggregators seamlessly to healthy instances. Production stability came from eliminating manual rebalancing entirely.

When Microservices Are Too Chatty: Hot Nodes and Object Churn

The first production version of our service topology pipeline exposed problems that load tests never hinted at. Three challenges dominated: consumer lag, uneven load distribution, and garbage collection pressure. Each turned out to be connected to the others.

Consumers Can’t Keep Up

Multi-region Kafka consumers began falling behind, with lag growing from seconds to hours. Instrumentation pointed to four issues: too few partitions for the consumer group size, small fetch sizes, undersized network buffers, and cross-region read latency.

We applied three fixes:

• Increased Kafka partitions to allow more parallel consumers
• Raised records per fetch to cut network round-trips, trading per-message latency for throughput
• Enlarged socket receive buffers beyond OS defaults

Lag dropped to under a minute at peak. The fix, however, exposed the next bottleneck: instances couldn’t process the higher ingest rate. Optimization in isolation only reveals the next constraint.

Aggregation by Destination Creates Hot Spots

Our initial design used consistent hashing so all flow logs for a given destination service landed on one “owner” instance. Grouping related data made sense for aggregation, but popular services like authentication generated orders of magnitude more traffic than typical endpoints. Some instances received 100x the flow records of others, spiking memory usage and triggering long garbage collection pauses. When an overloaded instance went down, its load shifted to peers, causing cascading failures.

Redistribution amplified the problem. A service called by 100 upstream services across 10 instances produced 10 separate aggregators on 10 different instances. The owning instance received and merged all 10, multiplying data volume during shuffling.

Profiling with async-profiler showed hot instances spending most of their CPU on GC while rapidly allocating aggregator objects. Memory pressure led to GC thrashing, which slowed processing and increased pressure further.

The three-stage pipeline— originally designed for proxy resolution—turned out to solve this too. Stage 1 aggregates raw flow logs into 5-minute windowed aggregators locally, discarding raw data quickly and reducing memory pressure. Stage 2 handles proxy resolution and provides an intermediate redistribution point; Stage 3 receives aggregators that have been compressed twice and distributed twice across two hashing operations. Even high-traffic services get spread across enough instances that no single node is overwhelmed.

The same investigation also showed gRPC was the wrong protocol for inter-stage communication. Replacing it with Server-Sent Events reduced resource consumption on both ends.

Immutability Costs Too Much on the Hot Path

Eliminating hot nodes reduced but didn’t eliminate heap pressure. GC logs showed pauses consuming more CPU than business logic; some instances still went down. Three factors contributed: objects lingered in the heap waiting for 5-minute aggregation windows, unnecessary type conversions created garbage, and Scala immutable data structures generated new objects on every update—at millions of records per second, this overwhelmed the collector.

We made four changes:

1. Released references faster: Optimized Pekko stream stages to aggregate and discard flow logs immediately.
2. Removed conversions: Routed aggregators directly between stages instead of converting to intermediate types.
3. Switched to mutable aggregators on the hot path: A pragmatic break from Scala convention, justified by measurements. Heap allocation dropped over 50% and GC pause times fell significantly.
4. Tuned time windows: Balanced data freshness against memory pressure.

GC pauses dropped from hundreds of milliseconds to tens. Best practices are starting points. At sufficient scale, deviation must be deliberate and measurement-driven—not applied indiscriminately.

Reactive Streams Require Active Mastery

Pekko Streams pipelines stalled without obvious errors, and debugging didn’t yield a stack trace pointing to the cause. Reactive streams invert control: downstream consumers pull from upstream producers, and .async boundaries create parallelism while complicating buffer sizing and demand signaling. Our initial overuse of .async added overhead rather than benefit.

We invested in team education, simplified stream graphs to mostly linear flows, and added metrics at boundaries to track buffer sizes, throughput, and backpressure events. The learning curve is real, but backpressure is worth it for systems that must absorb load spikes gracefully. The key is building a deliberate mental model and validating it through small experiments rather than assuming fluency from documentation alone.

Refining a Production System

Getting V1 into production solved the major architectural problems — Kafka lag, hot nodes, memory pressure — but operating at full scale surfaced a second wave of optimization opportunities. V2 is the story of turning a working system into one that can run reliably under sustained production load.

Memory Pressure Persists

Heap usage was still higher than desired after the V1 changes. Profiling revealed the cause: unnecessary object conversions between stages. Aggregators were being converted into full graph entities — with every property populated — before being routed to the next stage, even when that stage only needed the compressed aggregator state.

The fix was architectural: route aggregators directly through all stages, and only convert to final graph entities at Stage 3, immediately before persistence. Removing two intermediate conversion steps cut object allocation significantly, lowering heap usage and reducing GC pause frequency.

Serialization Inconsistency

Custom serialization logic for SSE messages produced intermittent errors that were difficult to reproduce. Different parts of the codebase used different serialization approaches, which compounded the problem. The team standardized on JSON encoding throughout the pipeline. JSON is slightly less efficient than binary serialization, but the overhead was negligible relative to other operations, and human-readable messages made debugging dramatically easier. Consistency alone eliminated an entire class of bugs.

Stream Configuration Tuning

The Pekko (Akka) stream configurations were suboptimal even after the reactive-streams learning curve. Some stages were over-parallelized, others under-parallelized, and the .async boundaries were not placed optimally. Through repeated profiling and experimentation, parallelism parameters and buffer sizes were tuned, and async boundary placement was refined. Adding monitoring at stream boundaries made bottleneck identification possible. The result was higher throughput and more consistent processing latency.

Uneven Graph Database Writes

Write distribution to the graph database was skewed: some partitions received heavy write traffic while others sat idle, causing uneven throttling and limiting overall write throughput. The solution had two parts. First, aggregators were batched rather than written immediately. Second, distribution logic across partitions was improved so that batched writes hit partitions more evenly. The outcome was steadier write throughput and better utilization of database capacity.

Enrichment at Aggregation Time

Topology nodes also carry context beyond the graph structure itself — application health status, ownership information, and other metadata from external sources. This enrichment is integrated at Stage 3, before persisting graph entities. Doing it at aggregation time rather than at query time avoids the performance cost of post-query joins and guarantees that every topology node has its full context available whenever it is queried.

Each V2 challenge followed a familiar pattern: production revealed an assumption that did not hold, profiling pinpointed the root cause, and a targeted fix improved a specific metric. Measure, hypothesize, validate, iterate — building at scale is a process of continuous learning, not a one-time correct design.

Time Travel: Reconstructing Topology History

One of the most powerful capabilities in the system answers temporal questions: What did the call graph look like when this incident happened? How have dependencies evolved over time? Traditional approaches fall short. Full snapshots at every point in time are prohibitively expensive to store. Event sourcing requires slow, costly replay to reconstruct a state.

The solution combines three mechanisms:

• Time-windowed aggregator snapshots: every aggregator stores startTs and endTs for its five-minute window. These immutable aggregators are persisted in the graph database keyed by (entity_id, timestamp), creating checkpoint states every five minutes.
• Property-level mutation tracking: the graph database keeps mutation history at the property level, storing only changed properties with their timestamps. This is far more efficient than copying entire entities and offers precision finer than the five-minute aggregation boundaries.
• Query-time reconstruction: when a historical query comes in, the system queries the mutation history API for the relevant time range, retrieves all mutations, and reconstructs the topology by applying them in order.

This design delivers efficient storage (compressed aggregator states plus sparse property mutations), fast retrieval (indexed mutation history, no log replay), and flexible analysis over arbitrary time ranges without pre-computing possibilities. Historical data can also be re-aggregated at query time using the same aggregator classes from the ingestion path. That enables ad-hoc groupings — by availability tier, business domain, or deployment cluster — without the storage cost of pre-computing every dimension.

Lessons for Distributed Systems at Scale

These challenges were specific to service topology, but the takeaways apply to distributed systems more broadly.

Scale Is a Qualitative Change

What works at 100 requests per second fails at 100,000. The increase is not linear, it is qualitative. Approaches that look fine at modest scale hit fundamental walls at extreme scale. In this project, immutable data structures created GC pressure at millions of allocations per second; single-stage aggregation failed catastrophically with power-law traffic distribution; standard gRPC turned out to be heavyweight for streaming aggregation at this volume.

The lesson is to be willing to abandon conventional wisdom when scale justifies it — but only on the strength of measurement, not speculation.

One Bottleneck at a Time

Distributed systems have cascading bottlenecks. Fix Kafka lag and a hot-node problem surfaces. Fix hot nodes and GC issues appear. Fix GC and serialization inefficiencies become visible. This is not failure; it is the nature of complex systems. Each optimization lifts throughput, which pushes against the next weakest point. The right approach is to prioritize by impact, fix the current bottleneck thoroughly, verify the resolution with measurement — then move on to the next one. Optimization at scale is continuous.

Distribution Is the Key to Scalability

A single aggregation point is always a bottleneck. Consistent hashing spreads load, but it cannot prevent concentration when the data itself is unevenly distributed — as it was here, with power-law traffic. The three-stage pipeline with graduated redistribution solved this problem. Load spreads across multiple distribution points at each stage, so even heavily skewed data cannot overwhelm a single instance. The general principle applies widely: use multi-stage processing with redistribution at each stage when data is skewed at scale.