The Netflix Cosmos Platform - Netflix TechBlog

Why Netflix Built Cosmos

The Media Cloud Engineering and Encoding Technologies teams at Netflix jointly run the system that ingests media from studios and partners and makes it playable across devices. The first generation shipped with the 2007 streaming launch; the second scaled up but was painful to operate; the third, Reloaded, has been stable and massively scalable for about seven years. But Reloaded was designed when a small team ran a constrained compute cluster for a single use case: the video/audio pipeline. Since then the developer count more than tripled, use cases expanded in breadth and depth, and scale grew more than tenfold. The monolith slowed feature delivery, required specialized knowledge to build and deploy anything new, mixed infrastructure code with application code, and centered on a data model that had become a liability.

Cosmos is the response: a platform for workflow-driven, media-centric microservices. Its first-order goals were preserving existing capabilities while improving observability, modularity, productivity, and delivery:

• Observability — built-in logging, tracing, monitoring, alerting, and error classification.
• Modularity — an opinionated framework for structuring a service that supports both compile-time and run-time modularity.
• Productivity — local development tools, including specialized test runners, code generators, and a command line interface.
• Delivery — a fully managed continuous-delivery pipeline. Merging a pull request gets code to production without manual intervention.

Along the way the team also improved scalability, reliability, security, and other system qualities.

Cosmos vs. a Typical Microservice

A typical microservice exposes an API with stateless business logic autoscaled on request load. The API provides strong contracts with peers while segregating application data and binary dependencies from other systems.

Cosmos keeps those strong contracts and the segregated data and dependencies, then adds multi-step workflows and computationally intensive asynchronous serverless functions. In a typical Cosmos service, clients send requests to an API layer (for example, a video encoder). A set of rules orchestrates workflow steps, and serverless functions power domain-specific algorithms. Functions are packaged as Docker images with their own media-specific binary dependencies, such as Debian packages. They scale based on queue size and can run across tens of thousands of containers. Individual requests can take hours or days to complete.

Two Axes of Separation

Cosmos separates logic in two dimensions. One axis divides logic among API, workflow, and serverless functions. The other separates application logic from platform logic. The platform API supplies media-specific abstractions, hiding distributed-computing details from application developers. A video encoding service, for instance, is built from scale-agnostic components — API, workflow, functions — that have no knowledge of the scale they run at. Those domain components sit on three scale-aware Cosmos subsystems that handle distribution:

• Optimus — the API layer mapping external requests to internal business models.
• Plato — the workflow layer for business-rule modeling.
• Stratum — a serverless layer for running stateless, computationally intensive functions.

All three subsystems communicate asynchronously through Timestone, a high-scale, low-latency priority queue. Each subsystem addresses a distinct concern and can be deployed independently via a purpose-built managed continuous-delivery process. That separation makes Cosmos services easier to write, test, and operate.

A Request Through the System

The screenshot above, from the Nirvana observability portal, shows a typical video encoder service request:

1. One API call to encode arrives, including the video source and a recipe.
2. The video splits into 31 chunks; 31 encoding functions run in parallel.
3. The assemble function is invoked once.
4. The index function is invoked once.
5. The workflow completes after 8 minutes.

Layering Services

Cosmos also supports decomposing services into layers. That modular architecture lets teams focus on their specialty and control their own APIs and release cycles.

The video service above is one of several involved in producing playable streams. Sibling services — inspection, audio, text, packaging — are orchestrated by higher-level services. The largest and most complex is Tapas, which takes studio sources and makes them playable on Netflix. Another high-level service, Sagan, handles studio operations like marketing clips and daily production editorial proxies.

When a new title arrives from a studio, a Tapas workflow orchestrates requests to run inspections, encode video (at multiple resolutions, qualities, and codecs), encode audio, generate subtitles in many languages, and package outputs for multiple player formats. A single Tapas request can therefore cascade into hundreds of requests to other Cosmos services and thousands of Stratum function invocations.

The trace above shows one top-level request trickling down through lower-level services. In that case the request took 24 minutes and involved hundreds of actions across 8 different Cosmos services and 9 different Stratum functions.

Workflow Rules, Not Workflow Steps

Plato ties Cosmos together, giving service developers a framework to define domain logic and orchestrate stateless functions and services. Optimus has built-in facilities to invoke workflows and inspect their state; Stratum generates strongly-typed RPC clients so invoking a serverless function feels straightforward.

Plato is a forward-chaining rule engine, chosen for the asynchronous and compute-intensive nature of Netflix's algorithms. Unlike a procedural workflow engine such as Netflix's Conductor, Plato makes it easy to build workflows that are "always on." As better encoding algorithms are developed, rules-based workflows automatically manage re-processing existing videos without triggering and managing new workflows. Any workflow can call another, which is what enables the layering described above.

Plato is multi-tenant, implemented with Apache Karaf, which cuts the operational cost of running workflows. Developers author and test rules in their own source repository, then deploy by uploading compiled code to the Plato server.

Workflows are specified in Emirax, a domain-specific language built on Groovy. Each rule has four sections:

• match — the conditions that must hold for the rule to fire.
• action — the code executed when the rule fires; this is where Stratum functions are invoked.
• reaction — code executed when the action succeeds.
• error — code executed when an error is encountered.

Each section typically first records the workflow's state change, then advances the workflow — executing a Stratum function or returning results. More detail is available in the feathercast presentation on Plato.

Keeping User-Facing Work Fast

Services like Sagan sit directly in the user path—an artist clipping a clip from a season of Money Heist shouldn't have to wait on infrastructure. In Stratum, latency is the sum of two parts: the time to run the work and the time to acquire the compute. For bursty workloads, the acquisition step dominates. A sudden spike in demand can stretch a normally quick process into a multi-week wait when everyone needs the same resource at once.

Stratum attacks function execution latency with several mechanisms:

1. Resource pools. Users can reserve compute for their own use cases; the pools are hierarchical, so teams can share capacity.
2. Warm capacity. Containers and other resources can be pre-provisioned ahead of expected demand to cut startup time.
3. Micro-batches. Borrowed from platforms like Apache Spark, this spreads the cost of a cold start over many invocations—10,000 functions might run once each on 10,000 containers, or ten times each on 1,000 containers.
4. Priority. Cosmos services typically size for normal peaks, not worst-case spikes. Setting priorities lets the most important work jump the queue when capacity tightens—end-users can set this, or service owners can enforce it at the API or workflow layer.

Throughput Over Speed

Tapas and similar services are throughput-sensitive. They burn millions of CPU-hours per day and care about completing many tasks over hours or days, not turning one task around quickly. Their SLOs are measured in tasks per day and cost per task, not tasks per second.

For these workloads, Stratum's serverless layer provides the critical SLOs. Built on top of the Titus container platform, Stratum offers "opportunistic" compute through flexible resource scheduling. A function invocation can cost less if it agrees to wait—say, up to an hour—before executing.

The Strangler Fig Approach

Replacing Reloaded—a large, legacy distributed system—was a high-risk jump. The team adopted the strangler fig pattern, letting Cosmos grow around the old system and eventually replace it wholesale, reducing the risk of a big-bang rewrite.

Running Since 2019

Cosmos has been in development since 2018 and in production since early 2019. About 40 services now run on it, with more expected. The team shares what it has learned so far.

Culture Over Architecture

Netflix's engineering culture favors personal judgement over top-down control. No one holds the title of Software Architect; everyone plays that role. Cosmos grew out of separate, independent efforts—Optimus, Plato and Stratum were each conceived on their own before coalescing into one platform. The application developers kept the team honest on API usability and developer productivity, and the partnership between infrastructure and media algorithm developers made the platform real. That kind of emergence doesn't happen in a command-and-control environment.

A Three-Layer Model

The pattern of "microservices that trigger workflows that orchestrate serverless functions" is powerful and covers most use cases. Some applications, though, are simple enough that the added complexity isn't worth the cost.

Platform Thinking

Shifting from one big application to a platform-plus-applications model was a mindset change for everyone. Application developers traded some flexibility for consistency and reliability; platform developers had to build empathy for user productivity and service levels. Friction was inevitable—each side occasionally felt the other wasn't listening. Honest retros helped, leading to dedicated tracks for crosscutting qualities like developer experience, reliability, observability and security.

Wins and Gaps

The initial goal was to let developers spend more time on business logic and less on infrastructure. The team is seeing gains land, with managed delivery, modularity, observability and developer support rated as the strongest qualities. Work remains on weaker spots: local development, resilience and testability.

Looking Ahead

In 2021, Cosmos takes on the majority of Reloaded's work, bringing more developers and much higher load. The programming model will evolve for new use cases. The goals are clear: easier to use, more resilient, faster and more efficient.