Maestro: Separating Visual Editing from Execution in Shopify Flow
Shopify Flow has long let merchants build automations for their stores. Behind the visual editor, a workflow is really a piece of executable logic, and as Flow has grown, so has the pressure on the architecture underneath it. The original Flow tightly coupled its editor and engine: both shared the same data structures, which meant neither could change independently. That coupling became a problem because the two sides have very different needs.
The editor demands a declarative language focused on usability, understandability, and interactivity, so merchants can express intent without worrying about how it runs. The engine, by contrast, benefits from a more imperative language, one that supports optimization, runs at scale, and guarantees at-least-once execution so workflows survive crashes. Still, the two sides must share a common type system so that user errors surface early and features like code completion and inline error reporting work inside the visual experience.
The team’s solution was to work incrementally rather than attempt a single sweeping migration. They first designed a new orchestration language, named Maestro, tailored to the engine’s needs. They built a horizontally scalable engine around it, then added a translation layer that converted legacy Flow workflow data into Maestro orchestrations. This allowed older workflows to run on the new engine without disturbing merchants; by BFCM 2020 nearly all workflows were executing there. Only then did the team tackle the editor, rolling out a new visual language that’s more flexible and expressive than the original, and translating it into Maestro orchestrations for execution. By early 2022 every Flow workflow had been migrated to the new editor and engine.
How Maestro Coordinates Work
Maestro is not a general-purpose language. It exists to orchestrate: that is, to define the order in which calls to host-language functions happen, and to record the data passed between them. For instance, imagine code that fetches customers from a remote service and then deletes them from a database. Maestro cannot implement the service request or the database operation itself — those belong to the host language — but it can coordinate those calls so the execution is fault-tolerant. The key benefit is that the state of each execution is captured precisely and can be made durable, letting you observe progress and resume from the same point if a crash occurs.
A simplified Maestro example gives a sense of the structure. First, it defines an object type Customer with its attributes. Then it defines three functions: fetch_customers performs a GET request and returns an array of customers; delete_customer simulates a database deletion via a print call from the standard library; and an orchestration function is the entry point that sequences the work. The sequence expression first calls fetch_customers and binds the results, then maps over the customers and calls delete_customer on each.
Functions wrap expressions: those with call expression bodies invoke other Maestro functions, while a function whose body is a primitive expression binds to registered host-language code. That mechanism is what yields to Ruby, or whatever host environment is running, to perform actual service requests, prints, and database operations.
Executing that orchestration with the interpreter, including the -c flag to dump checkpoints, shows what underpins fault tolerance. Checkpoints store enough state to tell what has already finished and what hasn’t. The first checkpoint, for example, records the service response containing the customers to delete. In practice, those checkpoints persist to durable storage such as Kafka, Redis, or MySQL. If the interpreter stops, you can restart it and point it at the existing checkpoints; it will skip expressions that already have checkpoints. Crash after deleting some customers but before finishing the rest, and the fetch won’t be rerun because its result is already stored.
This checkpointing mechanism delivers the at-least-once semantics Flow expects. The new engine is effectively a horizontally scalable pool of workers running the Maestro interpreter over events that come from Flow-generated orchestrations, using checkpoints both for crash recovery and to show merchants per-step status.
How Flow Rides on Maestro’s Type and Function Model
Consider a typical Flow workflow that fires when an order is created, checks the order for discount codes or a customer email, and conditionally tags the order and emails the store owner. A merchant using such a workflow performs four main activities: browsing the available tasks and data types, validating the workflow, activating it so events trigger execution, and monitoring runs.
To support browsing, Flow must model both the GraphQL Admin API types and the interfaces of the tasks on offer. It does so using Maestro types and functions, with extra decoration. Types mirror the structure of Maestro types but add field descriptions and other metadata. Most workflow types originate from APIs, so Flow has an automated pipeline that consumes those APIs and generates corresponding Maestro types. Additional types cover event payloads tied to triggers and the expected interfaces of actions.
Triggers, conditions, and actions are mapped onto Maestro functions. Actions map directly to functions that declare their parameters and return types; using an action in a workflow is just a call to that function. Triggers map to a data hydration function: a trigger event often carries only IDs, so the function accepts that minimal payload and issues API requests to load a fuller object — like fetching an Order from just an order ID — so downstream steps have the data they need. Conditions are currently a special case, translated into sequences of function calls based on the predicate the merchant defines. That arrangement lets Flow’s editor present a rich catalog of tasks and object attributes while keeping execution firmly in Maestro’s declarative, durable execution model.
Validating Workflows via Static Analysis
Before a workflow can run, Flow compiles it into a Maestro function whose parameter is the trigger data and whose body encodes the workflow's transitions and task configurations. In the example workflow, the sequence begins with a call to the trigger function to hydrate objects from event data. The condition's disjunction branches become calls to eq and ends_with, whose results feed an or call. A Maestro match expression then pattern matches on the result; a true outcome routes control flow to the sequence that invokes the action functions.
With this representation, Flow relies on Maestro's static analysis to validate the workflow function. The analyzer performs type checking, verifies variable scoping, and confirms object navigation is valid — for example, that order.customer.email actually exists. Any errors surface back in the Flow Editor, mapped to the corresponding workflow node.
Static analysis also produces symbol tables for each expression, indicating which variables are in scope and their types. The Editor uses these to offer per-step code completion and contextual suggestions. For instance, when configuring an Add order tags action, the Editor can guide users through the fields available on the objects in scope.
This transformation and validation cycle runs synchronously while a workflow is being edited, whether in the Flow Editor or through APIs. Because merchants are waiting on the result, the operation must stay very fast. The architecture mirrors how modern IDEs dispatch source code to a language service that parses it into a lower-level representation and returns diagnostics plus other static analysis data.
Activation, Optimization, and Deployment
Activating a workflow begins the same way as validation: Flow generates the corresponding Maestro function. From there, two additional steps occur.
First, Maestro performs static usage analysis. For each call to a primitive function, it computes which attributes of the returned type are actually consumed by later steps. The call to shopify::admin::order_created, for example, returns a tuple of Shop and Order, but the workflow may never reference order.customer.name. Hydrating unattributed values would be wasteful — and with recursive type definitions (an Order has a Customer who has Orders), it is impossible to know where to stop traversing the type graph. The usage analysis output is passed to the host function implementation at runtime, letting it tailor value computation, such as optimizing Admin GraphQL API queries.
Second, Maestro compiles the function. Optimizations strip everything the runtime does not need — type definitions and auxiliary functions that the workflow never calls. The resulting simplified function is packaged with the usage analysis result into an orchestration, which is serialized and deployed to the Flow Engine. There, the engine observes events and runs the Maestro interpreter against the orchestration.
Observing Runs with Checkpoints
During execution, the Maestro interpreter emits checkpoints at each primitive function call. These serve two purposes. The Flow Engine uses them when restarting the interpreter to honor at-least-once semantics for actions. The checkpoints are also sent back to Flow for the Activity page's execution list. Because each checkpoint carries detailed output from every primitive call, Flow can map it back to the originating workflow step.
The Run Log for a specific execution above — reachable from the Activity page — shows this in practice. Flow highlights which branch of the workflow executed and, within the condition disjunction, which branch evaluated to true at runtime. All of that visualization derives from interpreting checkpoints and correlating them to the workflow definition.
Planned Extensions
Maestro already powers Flow in production, but the roadmap includes several initiatives to expand its reach:
- Widening the expressiveness of Flow's workflow language by taking fuller advantage of Maestro's capability set. Planned additions include binding action results to variables for reuse in later steps, iteration support, and richer pattern matching.
- Applying further optimizations at deployment time, such as merging multiple Flow workflows into a single orchestration to avoid redundant hydration calls for the same incoming event.
- Leveraging the Maestro interpreter to preview and test workflows before activation, using checkpoints to surface results and verify assertions.
