From a Discontinued Repository to a Rebuilt Platform
Conductor is the workflow orchestration engine Netflix uses to stitch micro-services into reliable, observable business processes. The last public signal most observers had was the note on its GitHub repository: maintenance of the OSS project was discontinued so resources could go to the internal fork. Internally, development continued and accelerated, and it forced a ground-up rebuild. The growth is the clearest measure: crossing a billion workflow executions in a year was once a milestone, and that bar is now cleared roughly every quarter, with monthly volume still rising.
The Current Footprint
When Conductor was last covered publicly it ran a few hundred workflow definitions for a few dozen teams. It now backs roughly 200,000 workflow definitions owned by around 150 applications across Content and Studio Engineering, Ads, Games and other areas. In a typical month it executes on the order of 420 million workflows, and between 5 and 20 million on a given day. Teams add hundreds of new workflow types each month.
The Media Production Suite (MPS) illustrates the workload. An average Netflix title generates about 200 terabytes of original camera files, and MPS ingests, validates, transcodes and delivers that media. Its pipelines — Footage Ingest, Dailies, VFX Pulls, Conform Pulls — run as Conductor workflows. A dedicated orchestration layer, Studio Orchestrator, stitches together the platform services these pipelines depend on, including asset management, media encoding and storage, relying on Conductor for reliable execution and for the durability to recover from failures in long-running distributed work.
Two newer businesses use it just as heavily. Netflix Ads runs creative ingestion, advertiser-certificate management and privacy-preserving Data Clean Rooms as Conductor workflows. Netflix Games orchestrates a title’s path from a partner’s submission to the app stores: when a studio uploads a build, a pipeline ingests the binary, runs required validations such as iOS privacy-manifest checks, and drives App Store publishing across platform services and external stores.
Rethinking the Data Plane
The first generation leaned on Dynomite, with DynoQueues layered on top for queueing, and Elasticsearch for indexing. Two problem classes emerged with scale. Dynomite is an in-memory store — a clustering layer adding sharding and replication on top of Redis — scaled by adding memory and nodes, with each replica rack holding a full copy of the data. That suits a cache, not the durable record for millions of workflows a day. DynoQueues inherited those limits, and its zone-sharded design made task scheduling uneven. The engine compounded the issue: evaluation ran synchronously and mutated workflow state in several places, so competing updates could leave a workflow inconsistent with no automated repair.
The 2.x line replaced both core pieces. Cassandra became the primary datastore for execution data, providing elastic horizontal scaling and removing the scaling ceiling and zone-affinity constraints; large task inputs and outputs were offloaded to S3 to keep bulky data off the primary datastore. Timestone, Netflix’s high-throughput, low-latency priority queueing system, replaced DynoQueues and brought a delayed-queuing mechanism and a background repair process that could detect and recover workflows. Timestone later became the foundation for async evaluation and queue-depth-driven thread allocation. Indexing was also decoupled from the critical path via a Kafka-buffered, independently scalable indexer writing to Elasticsearch.
By Conductor 3.0 these threads converged: Dynomite fully sunset, Timestone as the queue, Cassandra as the datastore, a Kafka-fed indexer pushing to Elasticsearch with Iceberg added for long-term storage. The release introduced automated workflow repair and multi-cluster deployment, which lets workflows run in isolated contexts — for example, a dedicated context for the Media Production Suite, separate from the shared default and scaled independently.
Rebuilding the Evaluation Engine
By 3.0 everything around the engine was modern but the engine and its data model were largely original, and at this scale that was the bottleneck. Evaluating a workflow meant loading its entire state into memory to decide the next step, producing heavy memory pressure and slow evaluations as workflows approached tens of thousands of tasks. Metadata and task payloads shared a single Cassandra partition, creating oversized wide rows and leaving room for races when updates to the same task competed. Evaluation overhead, vertical-scaling pressure and occasional inconsistent state all needed fixing.
The rewrite separates workflow metadata from task and user data and gives every task its own record. The engine no longer loads the whole workflow graph; it evaluates against a lightweight blueprint and loads only the task data a decision requires. The 3.0 engine topped out around 2,500 tasks per workflow, while the rewritten engine scales to 30,000 — more than a 10x increase — with roughly a 40% reduction in p99 workflow-evaluation latency in production.
Concurrency Without Locks
A task can be written from more than one place at once. Consider a long-running task whose worker sends periodic in-progress updates to stay alive, then reports COMPLETED; a stray in-progress update arriving late after network reordering could revert the task to IN_PROGRESS. The first implementation used Cassandra lightweight transactions (LWT) to serialize writes, but mixing LWT with ordinary writes is a known anti-pattern — the two use different timestamping mechanisms, breaking the strict consistency Conductor needs.
The model therefore abandoned locking. Task state is split across a pending partition for in-flight tasks and a terminal partition written exactly once when a task reaches COMPLETED, FAILED or another terminal state. Evaluation reads both and reconciles them in the application layer: a terminal state always wins over a non-terminal one, and between two writes of the same kind, ordinary last-write-wins applies. In the race above, the COMPLETED write lands in the terminal partition and the late in-progress update loses on read.
The net result is reliable task lifecycle management, a lower memory footprint, faster evaluation and far higher concurrent throughput.
Async Evaluation
Evaluation still ran synchronously. An incoming update — a new workflow or a completing task — evaluated the workflow inline and had to hold a lock on it. Under load, concurrent updates to one workflow raced for the lock; the losers were deferred to a background repair process that could take several minutes to catch up, producing unpredictable tail latency at peak.
The fix decouples the trigger from the evaluation. Updates enqueue the workflow onto a Timestone exclusive queue, and a central asynchronous evaluator processes each workflow’s pending work in order. Because the queue is exclusive, only one worker evaluates a given workflow at a time and lock contention disappears. The gains are largest at the tail, which is where users feel the effect:
Unsuccessful lock-acquire attempts, which had spiked to around 2,700 per interval under contention, drop to essentially zero with async evaluation.
Controlling How Workflows Execute
Conductor historically ran as many instances of a workflow as you started, all at once. That default left a gap: when several instances act on the same entity — the same movie id, creative or script — there was no built-in way to serialize them, so teams built custom locks, queues and coordination logic themselves, adding complexity and operational risk to every workflow that needed it.
Run Strategies declare on a workflow definition how concurrent executions are handled. Parallel, the default, applies no restrictions. Sequential runs at most one workflow at a time per sequence key, with subsequent requests enqueued and started automatically in order as each run finishes. The strategy is declared on the definition and can be set in the Workflow SDK as a @RunStrategy annotation argument on @WorkflowMethod. The central idea is the sequence key: executions sharing a key run one after another, while executions with different keys still run in parallel. An ingestion workflow keyed on movie id processes each title strictly serially while running many different titles concurrently, with no custom locking.
Managed Tasks: Titus and Stratum
The traditional task was a polled worker: your application implements the worker, polls the Conductor server for work, runs it and reports back. That model is simple, language-agnostic and still the backbone for most workloads, but it means operating a worker fleet — standing it up, keeping it running, scaling it — even for bursty or rarely run work. Managed tasks are built-in task types where Conductor drives the compute directly, with no long-lived worker to host or scale. Two integrations carry most of this: Titus for containers and Stratum for serverless media-processing functions.
A TITUS task launches a Titus job to run a script — shell, Python or whatever the image packages — on Netflix’s container management platform. You supply a Docker image and entry point and specify the shape of the run, such as cpu and memory. The task starts IN_PROGRESS, Conductor monitors the job, and it transitions on the container’s exit behavior: exit code 0 marks the task COMPLETED, while a non-zero code, a timeout or a missing job marks it FAILED.
A STRATUM task invokes a Stratum function and collects its output. Stratum is the serverless function layer of Netflix’s Cosmos platform, built for media-processing workloads. The task takes a function name and an ordered argument list, stays IN_PROGRESS until the function finishes, then resolves to COMPLETED, FAILED or TIMED_OUT. A single workflow can mix polled workers, Titus jobs and Stratum functions — a long-lived worker for steady high-volume work, a container for an occasional heavy batch step, a Stratum function for media processing — moving the operational burden from the team to the platform.
Polling at Scale: Dynamic Thread Allocation
Faster evaluation is only half of throughput; the other half is how quickly workers pick up scheduled tasks. Conductor clients poll with worker threads, historically a fixed number per task type — predictable and starvation-proof, but static. During burst traffic, when thousands of instances of one task pile into a queue, a fixed thread count cannot ramp up. The symptom was distinctive: instance CPU stayed flat while queues backed up, and idle threads sat stranded on quiet queues unable to help the overloaded one.
The replacement rebalances threads across task queues continuously based on real-time queue depth, while guaranteeing every queue a floor. Each instance periodically polls Timestone for the current depth of every queue it manages, then splits its thread pool: a minimum allocation gives each queue a baseline, and a floating pool holds the remainder, distributed in proportion to each queue’s share of total load. A deep backlog draws a large slice; an idle queue falls back toward its minimum.
A multiplier ceiling prevents the rebalancing from becoming its own problem: a queue’s thread count can grow by at most a configurable factor per cycle, so a dramatic spike ramps in controlled steps and downstream services absorb the increase instead of being bombarded. Allocations are recomputed every polling interval, tracking shifting load in near real time while honoring per-queue minimums.
Two consequences follow. Because threads go where they are needed, each instance can safely run a much larger pool. And CPU utilization becomes an accurate signal of how busy an instance is, so autoscaling on CPU scales the cluster in response to real computational demand.
The Workflow SDK
Authoring once meant writing JSON: declaring each task, wiring outputs with ${taskRef.output.field} string templates, and on the worker side pulling inputs from a Map<String, Object> and casting to expected types. That pushed an entire class of errors to runtime and made large workflows hard to read. The Workflow SDK is a type-safe Java DSL for authoring Conductor workflows. You write Java; the compiler and IDE catch mistakes before deployment, and a Gradle plugin translates the code into the JSON definition the Conductor server understands.
Conductor’s visibility is preserved. Because a workflow is a declared graph rather than imperative code, execution can be inspected — every branch, loop and parallel fork lighting up live as an execution advances. The SDK keeps both halves: idiomatic Java at authoring time, and at execution time the same task-by-task visual graph, inputs and outputs, and at-a-glance view of where a workflow is stuck.
What It Removes
- No type safety. The JSON task input
"movieId": "${awesomeTask2.output.movieId}"is only a string template; the worker reads it withinputData.get("movieId")and casts. An upstream Integer cast to String throws a ClassCastException at runtime. A typo — declaringmovieIdbut readinginputData.get("movieid")— returns null and causes a NullPointerException downstream. None of it surfaces until the workflow runs. - Verbosity. Even a couple of tasks is wordy, and a real workflow’s JSON balloons into hundreds of difficult-to-review lines. Complex branching becomes a single large JSON blob no one wants to touch.
public class GetMovieWorker implements Worker {
@Override
public TaskResult execute(Task task) {
Map<String, Object> inputData = task.getInputData();
long movieId = (Integer) inputData.get("movieId"); // cast, fingers crossed
// more logic here
}
}
{
"name": "wrap_movie_workflow",
"version": 1,
"semver": "0.0.1",
"schemaVersion": 2,
"restartable": true,
"workflowStatusListenerEnabled": false,
"ownerEmail": "[email protected]",
"timeoutPolicy": "ALERT_ONLY",
"timeoutSeconds": 0,
"tasks": [
{
"name": "get_movie",
"taskReferenceName": "movie",
"type": "SIMPLE",
"inputParameters": {
"movieId": "${workflow.input.movieId}"
}
},
{
"name": "wrap_movie",
"taskReferenceName": "wrapMovie",
"type": "SIMPLE",
"inputParameters": {
"movie": "${movie.output}"
}
}
]
}
How the Java DSL Works
There are two core building blocks. @TaskMethod is a method that is executed at runtime and may contain arbitrary code; it takes and returns real types, so a getMovie task can accept a long movieId and return a Movie — no maps, no casts. @WorkflowMethod is a method that is not executed and cannot contain arbitrary code; it expresses the workflow as ordinary method calls (call getMovie(…), pass the result to wrapMovie(…)), and a Gradle build plugin parses it into a Conductor JSON definition at build time. The server never sees Java — it receives the same JSON as before.
@TaskMethod
public Movie getMovie(long movieId) { /* real worker logic */ }
@TaskMethod
public WrappedMovie wrapMovie(Movie movie) { /* real worker logic */ }
@WorkflowStub(taskNames = {"movieapp.getMovie", "movieapp.wrapMovie"})
public interface MovieWorkflows {
@WorkflowGenerated(name = "movieapp.getMovie")
Movie getMovie(int movieId);
@WorkflowGenerated(name = "movieapp.wrapMovie")
WrappedMovie wrapMovie(Movie movie);
@WorkflowMethod(name = "wrap_movie_workflow", ownerEmail = "[email protected]")
default WrappedMovie wrapMovieWorkflow(MovieInput input) {
Movie movie = getMovie(input.getMovieId()); // long in, Movie out (checked)
return wrapMovie(movie); // Movie in, WrappedMovie out (checked)
}
}
Code generation underpins this. In the @WorkflowStub annotation you declare the task and workflow names to use, and at build time the plugin generates their input and output types plus typed stub methods called inside a @WorkflowMethod. Those stubs let the method body compile against real signatures with full type safety, and because they are generated from names rather than shared code, teams can reuse one another’s tasks and workflows without sharing libraries.
Branching and fan-out use the Java constructs you would reach for anyway, parsed into the corresponding Conductor operators. The mental model is that you are describing a workflow graph in Java, not running Java, so only constructs that map cleanly are supported — if/else, switch, do-while, parallel forks, sub-workflow calls — and a few things must be expressed the SDK’s way. Iteration, for example, is always a do-while, since Conductor has no for or while tasks. A common shape kicks off work, polls until it settles, then branches on the outcome: a do-while wrapping an if/else.
@WorkflowMethod(name = "process_asset_workflow", ownerEmail = "[email protected]")
default void processAssetWorkflow(AssetInput input) {
JobStatus status = submitJob(input);
do { // loops are do-while
status = checkStatus(input.getJobId());
} while (status.getState() == "PENDING");
if (status.getState() == "SUCCEEDED") { // branch on the result
publish(input);
} else {
flagForReview(input);
}
}
Independent work runs concurrently with ParallelExecution.fork(…) plus allOf(…) — a FORK and JOIN — and any other @WorkflowMethod can be called as a sub-workflow by listing it in workflowNames:
@WorkflowStub(
taskNames = {"movieapp.getMovie", "movieapp.enrichMovie", "movieapp.getArtwork",
"movieapp.refreshCache", "movieapp.publishPage"},
workflowNames = {"movieapp.localize_page_workflow"})
public interface MoviePageWorkflows {
// stubs not shown for brevity
@WorkflowMethod(name = "prepare_movie_page", ownerEmail = "[email protected]")
default void prepareMoviePage(MoviePageInput input) {
// three independent branches, run in parallel
Fork<WrappedMovie> details = ParallelExecution.fork(() -> {
Movie movie = getMovie(input.getMovieId());
return enrichMovie(movie);
});
Fork<Artwork> art = ParallelExecution.fork(() -> getArtwork(input.getMovieId()));
Fork<Void> cache = ParallelExecution.fork(this::refreshCache);
ParallelExecution.allOf(details, art, cache); // FORK_JOIN and JOIN
WrappedMovie enriched = details.get();
Artwork artwork = art.get();
LocalizedPage localized = localizePageWorkflow(input); // sub-workflow call
publishPage(enriched, artwork, localized);
}
}
Shared work does not need reimplementation. Listing another team’s task or workflow in your @WorkflowStub is enough to call it like a local method or invoke it as a sub-workflow, and shared POJO types such as a Movie come along directly. The SDK is seeing rapid adoption and has become the primary choice for authoring new workflows at Netflix, with continued investment in IDE plugins, debugging and an integrated testing framework.
Adoption in Netflix Ads
Three Ads use cases show the range. Creative ingestion is a heavily branching pipeline where the SDK converted a sprawling JSON blob into type-safe, testable code, and a sequential run strategy removed a duplicate-processing race by serializing work per creative. Advertiser certificate management automates a previously manual, hard-to-scale process across several system integrations, with downstream automation when certificates expire. Data Clean Rooms — privacy-preserving environments where Netflix and a partner run joint analysis on shared data — were built on the SDK from day one; provisioning is a multi-day sequence (create at a vendor, wait for the partner to join, activate, eventually archive) that previously relied on hand-rolled async polling. Re-authored as Conductor workflows, each phase became a durable, observable state machine with automatic retries, state that survives restarts, and long partner waits that yield rather than hold a thread. The common thread: teams lean on Conductor’s reliability and visibility instead of hand-rolling retries and distributed locking.
What’s Next
- Scale. As Netflix expands into live, games and podcasts, workflow load could grow 5x in the coming year; the engine work above is what gives confidence that this can be absorbed without re-architecting again.
- Developer experience. The SDK now covers all task types, shifting focus to the surrounding experience: a richer integrated testing story so workflows can be validated against real worker behavior, deeper agent tooling such as skills, plugins and MCPs, and safe schema evolution via a schema registry for shared tasks and workflows with guardrails blocking non-backward-compatible changes.
- Execution control. Sequential is only the first run strategy. Richer policies are planned, including running N at a time per key, idempotent de-duplication, keep-first and keep-latest, and combinations of these. Managed tasks will keep expanding the ways Conductor runs compute directly beyond today’s container and serverless integrations.
The direction is consistent: keep overhead low, keep authoring safe and simple, and keep widening the range of work a single workflow can reliably coordinate.
