How Netflix Accurately Attributes eBPF Flow Logs - Netflix TechBlog

From IP Address to Workload Identity

Netflix’s FlowExporter sidecar uses eBPF and TCP tracepoints to capture TCP socket state changes and generate flow logs as sockets close. These logs contain IP addresses, ports, timestamps, and socket statistics, and the fleet produces about 5 million records per second. In cloud environments, however, IP addresses get reassigned as instances come and go, so the raw logs offer little insight until each IP is tied to a workload identity. The FlowCollector backend service performs that attribution and forwards the enriched flows to Netflix’s Data Mesh for stream and batch processing.

The challenge is that IP-based attribution is error-prone. Netflix initially relied on Sonar, an internal IP tracking service that emits an event whenever an IP is assigned or unassigned in an AWS VPC. FlowCollector consumed that event stream and used it to attribute flows in real time. But delays and failures are unavoidable in distributed systems, and a delayed reassignment event could cause FlowCollector to attribute an IP to the wrong workload. Event timestamps could also be inaccurate depending on how they were captured.

Misattribution made the flow data unreliable. Users depend on flow logs to confirm workload dependencies, but incorrect attributions created confusion, and the problem got worse for large, frequently changing services. As a stopgap, FlowCollector held flows for 15 minutes before attribution to wait out delayed events. That reduced misattribution but did not eliminate it, and the extra latency made the data far less useful for real-time analysis. Even one misattributed flow can produce an incorrect dependency, so Netflix redesigned the attribution approach entirely.

Attributing the Local Side

Every socket has two addresses: local and remote. The original system treated them the same way, but the local address is actually easier to attribute because it belongs to the instance where FlowExporter is running. FlowExporter can determine the local workload identity from its own environment and label the flow before sending it upstream.

For workloads on EC2 instances, this is straightforward: Netflix’s Metatron provisions identity certificates to each instance at boot, and FlowExporter reads them from disk. Container workloads on Titus are more complicated. FlowExporter runs at the container host level, where many containers with different identities share the host. When a kernel tracepoint fires, the socket could belong to any of them, or to the host itself.

To resolve that, Netflix uses IPMan, its container IP assignment service. An IPManAgent daemon on each host writes an IP-to-workload-ID mapping into an eBPF map when containers launch. FlowExporter’s eBPF programs can then look up the owner of a socket’s local IP in that map.

IPv6-only Titus containers introduce another wrinkle. Netflix developed a mechanism that lets IPv6-only containers talk to IPv4 destinations without NAT64, by intercepting connect syscalls and swapping in a socket that uses a shared IPv4 address on the container host. The kernel therefore reports the same local IPv4 address for sockets from different workloads. To disambiguate, Titus writes a mapping of (local IPv4 address, local port) to workload ID into an eBPF map at the moment the syscall is intercepted, and FlowExporter uses that map for correct attribution.

Attributing the Remote Side

Once the local address is attributed, the remote side becomes tractable. Each flow now carries the local IP, the local workload identity, and the connection start and end timestamps. As FlowCollector processes these flows, it learns the time ranges during which each workload owned a given IP. If a flow arrives with local IP 10.0.0.1 belonging to workload X from t1 to t2, FlowCollector now knows that 10.0.0.1 was X’s during that window. Netflix uses Amazon Time Sync fleet-wide, so the timestamps FlowExporter captures are reliable.

FlowCollector runs as a cluster of nodes, and every node must be able to attribute any remote IP. Each node keeps an in-memory hashmap from IP address to a list of non-overlapping, time-sorted ownership ranges. Nodes populate this map from the flows they receive and share what they learn through a Kafka-based broadcast mechanism. A flow’s start timestamp determines which time range applies, and if no range covers that moment, FlowCollector retries after a delay before giving up. Some flows may remain unattributed if flow records are lost or broadcasts are delayed, but no misattribution occurs.

This design succeeds because flows act as continuous heartbeats tied to reliable IP ownership time ranges. A few delayed or lost heartbeats do not break attribution, whereas the old approach depended on discrete assignment events and assumed an IP stayed with a workload until told otherwise — sometimes hours or days later.

One operational detail: FlowCollector cannot attribute a remote IP immediately because it needs the latest observed ownership ranges, which may still be in transit from the remote workload’s FlowExporter. Since FlowExporter reports in one-minute batches, FlowCollector stores flows on disk for a minute before attributing remote addresses. That one-minute delay is a significant improvement over the 15 minutes required by the old method.

The approach is also cheap to run. Lookups are in-memory, the state can be rebuilt quickly at startup, and no persistent storage is required. Netflix handles the entire fleet’s 5 million flows per second with 30 c7i.2xlarge instances.

Regional Routing for Cross-Region Flows

Netflix’s cloud workloads span multiple AWS regions. To keep reporting efficient, a FlowCollector cluster runs in each region, and FlowExporter agents send their flows to the local collector. A collector can therefore assume that the local IP address in any flow it receives belongs to its own region.

Flow attribution depends on an IP address time-range map broadcast among collectors. To limit cross-region traffic, that broadcast is confined to collectors within the same region, meaning the map only contains local IPs. When a flow’s remote IP belongs to another region, the receiving collector resolves the destination region from a trie built from all Netflix VPC CIDRs, then forwards the flow to collectors in the appropriate region. This targeted forwarding is practical because only about 1% of flows are cross-regional; broadcasting address updates across all regions would be wasteful for such a small share of traffic.

Handling Non-Workload IPs

Flows that terminate at AWS Elastic Load Balancers (ELBs) present a different attribution problem. FlowExporter cannot run on ELBs, so a collector cannot identify those IPs from observed flow traffic. ELB IP addresses are instead attributed using change events from Sonar, Netflix’s crawler for AWS resource state. The Sonar stream can carry delayed or imprecise timestamps, but this is acceptable because ELB IP reassignment is rare, so the risk of misattribution is low.

Validation via Zuul

Proving that the new attribution method eliminates misattribution is not straightforward. Flow logs are meant to be the authoritative record of service dependencies, so there is no separate ground truth to compare against. To get around this, Netflix looked at a large service with dependencies that can be established independently: the cloud gateway Zuul.

Zuul is a strong test case for several reasons. It handles all cloud ingress traffic, giving it a very large footprint, which matters because misattribution is most visible among services with many instances. It also has a substantial set of downstream dependencies, and those dependencies can be derived from its routing configuration — no flow logs required. Comparing flow logs against that routing configuration over a two-week window produced a clean result: no misattributed flows. Under the old attribution scheme, roughly 40% of Zuul’s reported dependencies were wrong.

Outcome

With misattribution resolved, eBPF flow logs are now a reliable source of fleet-wide topology and network health data at Netflix. That opens possibilities in dependency auditing, security analysis, and incident triage, and gives engineers a more accurate picture of how distributed systems evolve.