From Library Stacks to a Shared Data Plane

Netflix’s migration to AWS began in 2008, well before the cloud-native tooling landscape existed. The CNCF wasn’t formed until 2015, so the platform team had to build its own foundation for inter-process communication (IPC). The environment demanded client-side load balancing: nodes are ephemeral, and services must react quickly to topology changes and route around failures. That approach also avoids a centralized load balancer as a single point of failure in the request path.

Two core components emerged from that era. Eureka handles service discovery, and Ribbon (known internally as NIWS) provides the client-side IPC logic with resiliency features. The interface they present is intentionally simple: a service identifies a destination by a Virtual IP (VIP) name for insecure traffic or a Secure VIP (SVIP) name for secure traffic, optionally enabling features like retries or circuit breaking on top of sensible defaults. The client then fetches the matching set of IP and port pairs from Eureka.

Press enter or click to view image in full size

If Eureka itself fails, services can still communicate using cached host information, albeit with increasingly stale endpoint data. The system degrades gracefully rather than halting traffic entirely.

Why Centralize IPC Logic

The original architecture held up for over a decade, but the IPC ecosystem has grown more complex on three fronts. Netflix now runs a mix of plain REST, GraphQL, and gRPC traffic. The environment is no longer Java-only; it includes node.js, Python, and various OSS and commercial software. And the feature set has expanded to include adaptive concurrency limiting, circuit breaking, hedging, and fault injection.

Maintaining feature parity and identical behavior across all those language-specific clients is a losing battle. A service mesh offers a way out: move the IPC feature set into a single, well-tested proxy implementation, and reduce per-language clients to thin shims that only know how to speak to the local proxy.

Envoy fits the bill. It is battle-tested at scale in production, includes many critical resiliency features, and offers extension points for custom functionality. Crucially, Envoy supports dynamic configuration via a central control plane, which preserves the operational benefits of a central load balancer without putting one in the request path.

Reusing the VIP Abstraction in Envoy

Two constraints shaped the migration strategy. First, keep the existing interface—the VIP/SVIP abstraction had to remain intact for backwards compatibility. Second, automate as much of the migration as possible to make it seamless for service owners.

Envoy already provided the right building blocks. A VIP maps naturally to an Envoy Cluster, and clusters can be pushed to proxies via the Cluster Discovery Service (CDS). Hosts within a cluster become Envoy Endpoints, managed through the Endpoint Discovery Service (EDS). The existing discovery abstractions could therefore live on inside the mesh, letting existing IPC clients continue to function under the hood.

From Static Config to Runtime Lookup

Envoy’s requirement that all clusters be declared in a proxy’s static config quickly became a bottleneck during Netflix’s mesh migration. A single service can depend on dozens of upstream clusters, and that set shifts as new features are added or architecture evolves. Manually maintaining these definitions was not realistic at scale:

  • Service owners rarely know the full list of clusters their code talks to, especially when libraries transitively call other services or when operational dependencies like telemetry and logging are involved.
  • Generating config from an existing call graph works for current services but fails for new services or new upstream connections that haven’t appeared in traffic yet.
  • Pushing the complete set of clusters to every proxy was mathematically infeasible—millions of endpoints per proxy was far outside acceptable memory and bandwidth limits.

The alternative was to fetch cluster definitions lazily at runtime. Netflix partnered with Kinvolk and the Envoy community to implement On-Demand Cluster Discovery (ODCDS), which lets a proxy resolve a cluster the first time it attempts a connection, rather than requiring it in advance.

How the On-Demand Flow Works

Netflix’s control plane already implemented the Envoy XDS services, so the missing piece was wiring that to Eureka, which remains the source of truth for VIPs and instance health. VIPs and SVIPs are modeled as separate CDS clusters (e.g., myservice.vip and myservice.svip), and individual hosts within a cluster become EDS endpoints. This preserves existing Eureka abstractions and lets IPC clients like Ribbon move to mesh with minimal changes.

The request path for the first connection to a new cluster is:

  1. A client request arrives at Envoy.
  2. Envoy extracts the target cluster from the Host / :authority header (configurable, but that’s the default). If the cluster is already Known, skip to step 7.
  3. The cluster is unknown, so the request is paused.
  4. Envoy calls the CDS endpoint on the control plane, which returns a cluster definition customized for that service based on its config and Eureka registration.
  5. The new cluster triggers an EDS pull for its endpoints, filtered by Eureka status for the VIP or SVIP.
  6. The paused request resumes.
  7. Normal processing continues: load balancing across the discovered endpoints and dispatch.

The whole sequence completes in a few milliseconds, and only on the first request to a given cluster. After that, the cluster exists as if it had been in the original config.

Press enter or click to view image in full size

Trade-Offs and What’s Next

The primary downside is the added latency on a cold cluster. For workloads that require minimal first-request latency, teams must either predefine their clusters or prime connections before traffic starts. Netflix has also explored pre-pushing clusters at proxy start-up based on historical patterns. For the majority of services, the operational simplicity of auto-discovery outweighs the few extra milliseconds on the first hit.

The mesh work is still in its early stages. After porting adaptive concurrency limiting to Envoy, the team is eyeing incremental EDS as the next collaboration area. EDS updates are the largest source of control-plane and proxy pressure, so streaming only endpoint deltas would materially reduce load on both sides.