100X Faster: How We Supercharged Netflix Maestro’s Workflow Engine

From Seconds to Milliseconds: Rebuilding Maestro's Engine

Maestro, Netflix's workflow orchestrator, was built to scale horizontally and currently manages millions of executions daily across hundreds of thousands of jobs. That scalability goal was met, but as adoption grew, new use cases emerged. Sub-hourly schedules, ad hoc queries, and iterative development cycles all exposed a problem: the engine added a noticeable overhead, often ten seconds or more per operation. For developers working through repeated test-and-fix cycles, that latency became a real drag on productivity.

The root cause wasn't the API layer or the step runtime, which handled integrations with compute engines efficiently. The bottleneck was the internal flow engine layer. Built on the deprecated Netflix OSS Conductor 2.x, it added significant delay, and its lack of strong guarantees — like exactly-once publishing — led to race conditions that caused stuck jobs or lost executions.

Our response was to rewrite the internal flow engine from scratch. The new design is lightweight, has minimal dependencies, and cuts end-to-end overhead by roughly 100X, bringing overall processing delays down from seconds to milliseconds. The rewritten engine is now part of the open-source Maestro project.

Why the Old Architecture Felt Slow

The previous Maestro system was split into three layers. The API and step runtime layer handled user interactions and integrations with platform services, and it worked fine without adding notable overhead. The middle engine layer managed the lifecycle of workflows and steps, translated user-defined graphs into parallel flows of sequential tasks, and wrote state to the database. This layer performed acceptably, but it also had edge cases where a step could be picked up by two workers concurrently, leading to race conditions, because the underlying flow engine and distributed job queue couldn't provide strong guarantees.

The real problem was the foundational internal flow engine. It did only two things: it called task execution functions at set intervals, and it started the next task in a sequentially chained flow. But despite this simple purpose, it relied on its own dedicated database tables and a distributed job queue. That infrastructure introduced the bulk of the overhead — a few seconds to tens of seconds of delay on every operation — and it could not guarantee that a task was processed exactly once.

Evaluating the Alternatives

Before committing to a rewrite, we considered two other paths.

The first option was upgrading to Conductor 4.0, which addressed some overhead and offered general improvements over the 2.x line. But there was a catch. The older Conductor version included a final callback capability that Maestro relied on to synchronize the Conductor state machine with Maestro's own engine state. That capability had been contributed specifically for Maestro and had been dropped from Conductor 4 because no other use case depended on it. Porting it forward was possible but would have meant maintaining a state machine in two systems indefinitely.

The second option was adopting Temporal as the flow engine. Temporal is a robust control-plane orchestration system, and it works well when you need inter-process orchestration across services. But that means every DAG execution step would call an external service to interact with the flow engine. With Maestro processing over a million tasks per day, many of them long-running, coupling our core execution path to external service calls felt like an unnecessary reliability risk.

Both alternatives kept the fundamental problem intact: Maestro would still be parsing and evaluating its workflow graphs through a full-featured DAG engine that was not built for our specific, narrow needs. Instead, we chose a third path and rewrote the flow engine entirely, keeping only the two functions that mattered for our workload: executing tasks on schedule and advancing sequential flows.

What the New Engine Guarantees

The rewritten internal flow engine drops the external job queues and database dependencies in favor of internal ones. It is designed to be highly performant, efficient, scalable, and fault-tolerant. More importantly, it provides concrete guarantees that eliminate the race conditions that plagued the previous version:

• A single step is executed by only one worker at any given time.
• Step state is never rolled back.
• Steps always eventually reach a terminal state.
• The internal flow state stays eventually consistent with the Maestro workflow state.
• External API calls and user actions cannot cause race conditions on the workflow execution.

These guarantees were not practical with the previous design, which synchronized state across two separate engines and their own databases. By owning the complete state machine internally and simplifying the architecture, we removed the synchronization overhead and the associated failure modes, which is what unlocked the dramatic performance improvement.

In-Memory State Core

The redesigned flow engine keeps workflow state entirely in memory and treats the Maestro engine's database as the authoritative source for workflow and step states. During bootstrap, the flow engine reconstructs its in-memory state from the database, which simplifies the architecture considerably. The previous design required reconciling multiple databases—Conductor's tables alongside Maestro's—which could lead to race conditions and rare orphaned job statuses.

The new engine operates on in-memory flow states using a write-through caching pattern: updates to workflow or step state in the database are mirrored immediately in the in-memory state. If in-memory state is ever lost, the flow engine rebuilds it from the database, providing eventual consistency and eliminating race conditions. This results in lower latency, higher throughput, and a cleaner separation of concerns with the in-memory view kept consistent with the database.

Collocated Execution and Flow Groups

Performance gains come largely from collocating flows and their tasks on a single node for their entire lifecycle. States stay in one node's memory without persisting to the database on every step. This stickiness improves performance dramatically, but it affects scalability because tasks are no longer reassigned to any available worker across the cluster each polling cycle.

To keep horizontal scalability intact, Maestro now partitions running flows into groups. Each flow engine instance maintains ownership only of groups, not individual flows, which reduces heartbeat and reconciliation overhead. A flow group actor claims a group of flows, and child flow actors manage each flow's lifecycle. If ownership is lost—due to node failure or a long JVM GC pause—another node claims the group and resumes execution by reconciling internal state from the Maestro database.

Flow partitioning uses a stable ID assignment method to consistently map workflow IDs to group IDs. Parent flows pass the maximal group number down to all child flows, enabling child flows such as subworkflows or foreach iterations to recompute their own group IDs. This guarantees that a parent can always locate the group of any child flow using only workflow identifiers. The group size is configurable per node, and the maximal group number can change during engine execution without affecting existing workflows.

Queue Replacements

Both external distributed job queues were replaced with internal alternatives. The flow engine's queue is a simple in-memory Java blocking queue with no persistence required—state can be rebuilt from Maestro during reconciliation.

The Maestro engine now uses a database-backed in-memory queue providing exactly-once publishing and at-least-once delivery guarantees. Modeled after the transactional outbox pattern, a row inserted into maestro_queue in the same transaction that updates Maestro tables is pushed immediately to a queue worker on the same node upon commit, eliminating polling latency. After successful processing, the worker deletes the row. A periodic sweeper re-enqueues any rows whose timeout expired, picking up work from stalled workers or failed nodes.

This design handles failures gracefully: failed transactions roll back both data and message atomically, and workers or nodes that fail after commit are covered by the timeout-based retry. Each event type is assigned a queue_id to partition messages and avoid contention under high load.

From Workers to Actors

The previous architecture used a shared-nothing stateless worker model. Task identifiers were placed on distributed queues; a worker picked one up, loaded the full workflow state from the database, executed the task, wrote results back, and re-enqueued with a polling delay. This worked but introduced substantial overhead from polling intervals and repeated state loads. Complex workflows decomposed into multiple flows could span many polling cycles, adding up to roughly ten seconds of latency in the worst cases.

The model also lacked strong execution guarantees. Because the distributed queue provided at-least-once semantics, tasks could be dispatched to multiple workers. Two workers might pick up the same task after a GC pause: one marks it completed and unblocks downstream steps, while the other, holding stale state, resets the task to running—leaving the workflow in conflict.

The new design uses a stateful actor model with all states kept in memory. Tasks of a workflow are collocated within the same Maestro node, so states live in the same JVM. Java 21 virtual threads implement each entity—workflow instance or step attempt—as an actor, with virtual threads chosen for their lightweight nature and fit with state machine transitions.

The actor-based flow engine is fully event-driven; actions execute immediately when events arrive, eliminating polling delays. A wakeup mechanism maintains compatibility with existing polling-based logic. Flow actors and their child task actors stay in the same JVM, communicating through in-memory queues.

• Engine startup or reconciliation loads Maestro workflow and step instances from the database and transforms them into internal flow and task state, kept in JVM memory until eviction.
• Each entity spawns its own virtual thread actor that handles all its updates, ensuring thread safety without distributed locks.
• Each actor holds in-memory state, a thread-safe blocking queue, and a state machine that advances state transitions.
• Flow actors manage their child task actors hierarchically; same-JVM colocation delivers locality benefits while still permitting relocation when required.
• Events wake virtual threads by pushing messages onto the actor queue, enabling event-driven behavior alongside polling.
• A reconciliation process maps Maestro data models to internal flow data.

Thread Safety and Execution Guarantees

Virtual threads are suited for internal state transitions but can deadlock when executing user-provided logic or complex step logic depending on external services. Maestro separates engine execution from task execution: a separate worker thread pool (not virtual threads) runs step business logic such as launching containers or external API calls. Flow and task actors wait indefinitely on thread pool executor futures without performing actual execution, avoiding deadlock while benefiting from virtual threads.

Each engine node claims a group and updates its group generation ID upon bootstrapping. Group actors update the generation IDs of all owned flows while rebuilding internal state. Whenever a new flow is created, the group actor verifies the database generation ID matches its in-memory copy, rejecting creation with a retryable error if mismatch occurs. This guarantees a single actor executes any flow or task at a given time, with non-rolling-back state that eventually reaches a terminal state.

The engine now supports both event-driven execution and polling-based periodic reconciliation, enabling extended polling intervals at low cost—with event delivery relaxed to at-most-once.

Testing the New Engine at Scale

Rolling out a new DAG engine across hundreds of thousands of Netflix data processing jobs demanded a testing strategy that went beyond static unit and integration tests. The team built a dedicated test framework for Maestro that samples real production workflows but strips away external side effects like data reads and writes. This lets engineers exercise workflow graphs of various shapes and sizes — parallel executions, for-each jobs, conditional branching, parameter passing — without touching live systems.

The framework works in two phases. First, it caches production workflows: successful instances are pulled from a historical Maestro feed table, with run parameters, initiator, and instance IDs organized into an instance data map. YAML definitions and subworkflow IDs are fetched from S3, and everything is cached for replay.

In the second phase, cached definitions and instance data are loaded and pushed. Notebook-based jobs are swapped for custom notebooks, and certain job types (vanilla container runtime jobs, templated data movement jobs, signal triggers) are converted to a no-op job type or skipped. Abstract job types like Write-Audit-Publish, which expand into multiple DAG nodes at execution time, are auto-translated into several custom notebook job types.

Sub-workflows get special handling: in the parent workflow, each sub-workflow is replaced with a no-op placeholder so the overall topology is preserved without executing child side effects. Each sub-workflow is then run separately as its own top-level workflow to exercise its actual steps. The custom notebooks internally compare all passed parameters per job, and workflow instances are monitored until termination, with a failure report emailed out.

Cutover Strategy and Rollback Safety

The rollout philosophy was simple: a workflow, from its root instance, had to live entirely in either the old or the new engine. No mixed operations. A parallel infrastructure hosted the new flow engine, and Maestro's orchestrator gateway API masked all routing logic from users. Workflows could initially opt in via a system flag, letting the team observe behavior and build confidence. Scaling traffic up on the new stack in direct proportion to scaling down on the old one kept the dual-infrastructure cost negligible.

Once confidence grew, the team moved to a percentage-based cutover. If the new engine sustained a failure, rollback meant removing the workflow from the new engine's database and restarting it in the original stack. The cost: failed workflows had to restart from the beginning, recomputing previously successful steps to guarantee all artifacts came from a consistent engine.

Maestro's 10-day workflow timeout enabled a user-free migration. Existing executions either completed or timed out; on restart (after failure or timeout) or new instance trigger (after success), the workflow was picked up by the new engine. This gradually drained traffic from old to new with no user involvement.

Challenges Encountered

Around 50 workflows with defunct or incorrect ownership information got stuck. In some cases, a backlog of queued instances behind a stuck instance created a race condition: terminating the old instance would immediately start a new one on the old engine, keeping it there indefinitely. The team proactively contacted users to negotiate manual stop-and-restart windows.

A more significant lesson involved configuration management. Alerts, system flags, and feature flags configured for one stack were not always mirrored in the other. A partner team's Python migration tool, which analyzes workflow configurations dynamically, silently skipped ~40 workflows because a required feature flag was missing in the new engine stack. This resulted in incorrect Python version configurations. The issue was quickly remediated, but affected workflows had to be restarted and verified — and it exposed a limitation in the test framework, since runtime configuration based on external API calls was never exercised in simulated runs.

Migration Results and Payoff

Over 60,000 active workflows, generating more than a million data processing tasks daily, were migrated with almost no user involvement. Measurements confirmed the architectural gains: step launch overhead dropped from around 5 seconds to 50 milliseconds, and workflow start overhead (incurred once per execution) went from 200 milliseconds to 50 milliseconds. Aggregated over a million daily step executions, that translates to roughly 57 days of flow engine overhead saved per day. Users see snappier workflow status and the same infrastructure handles greater task throughput.

The internal maintenance burden also fell. The new flow engine's simplified database components allowed the team to delete nearly 40TB of obsolete tables tied to the stateless engine, and internal database query traffic — previously a major source of alerts — dropped by 90%.

What the Rewrite Delivered

The redesign, centered on a stateful actor model, cut per-operation overhead from seconds to milliseconds — a 100X improvement. Beyond raw speed, the architecture brought several benefits:

• Performance at the operation level matters, even in a system built for scale. Individual step latency has an outsized effect on user experience.
• Simpler architecture wins. Fewer dependencies improved reliability and maintainability alongside speed.
• Strong execution guarantees eliminate the race conditions and edge cases that previously demanded manual intervention.
• Locality pays off. Collocating related flows and tasks in the same JVM drastically cuts engine overhead.
• Java 21's virtual threads enabled an elegant actor-based implementation with minimal code complexity and added dependencies.

The project is open source. The code lives in the Maestro GitHub repository, where questions and issues are welcome via GitHub issues.