Metaflow Meets AWS Step Functions

Netflix has released a new integration that lets Metaflow users schedule production workflows with AWS Step Functions. The integration requires no changes to existing Metaflow code and provides a highly available, scalable scheduling layer that is maintenance-free.

Metaflow, the data science framework Netflix open-sourced in December 2019, was designed around the idea of independent abstraction layers. While excellent solutions exist for each layer of the typical data science stack, stitching them together has historically been a challenge for data science projects. Metaflow acts as a substrate that integrates these layers into an easy-to-use productivity tool, without pretending the layers don't exist. Problems are solved by people, not by tools, so hiding the stack entirely would be counterproductive—especially when things fail.

This integration addresses the job scheduler layer, sitting between the architecture layer (which defines code structure) and the compute layer (which defines how code executes). Since Metaflow's open-source release, users have asked how it compares to workflow schedulers and how workflows should run in production. Both questions have the same answer: Metaflow is designed for use alongside a production-grade job scheduler. AWS Step Functions is the first such scheduler with an open-source integration.

Unbundling the DAG

Metaflow asks users to organize their work as a Directed Acyclic Graph of compute steps—a natural abstraction for data science workflows. At this level, the DAG says nothing about which code gets executed or where it runs; it only describes how the data scientist wants to structure their code.

Many existing systems require tight coupling between layers of the data science stack, often due to infrastructural limitations that predate the cloud. Users may need to write modeling code in a custom DSL, which must execute with a built-in scheduler that is tightly coupled to a specific compute layer like an HPC cluster.

This tight coupling works for specific use cases, but Metaflow supports hundreds of real-life data science applications—from natural language processing and computer vision to classical statistics with R. A tightly coupled stack of what-how-and-where cannot serve all these use cases well. A user might want a compute layer optimized for computer vision for certain steps in their workflow, for example.

Metaflow unbundles the DAG into separate architecture (what), scheduler (how), and compute (where) layers. Users write code in languages they know, using the rich R and Python data science ecosystems. Metaflow packages that code for the compute layer, so users don't write Dockerfiles. The scheduler layer executes individual functions using the compute layer. From the data scientist's perspective, idiomatic modeling code simply runs—even at massive scale.

Architecture: The What

Metaflow provides an opinionated syntax and utilities for crafting workflows in Python and R. It handles data flow and state transfer through the graph, independent of the scheduler. The user-facing API carries strong backwards-compatibility guarantees, so code written today will still schedule and execute as underlying layers evolve.

Scheduling: The How

The scheduler's sole responsibility is orchestrating steps in topological order, ensuring each step completes before its successors run. The scheduler doesn't care what code executes. Metaflow comes with a built-in local scheduler for development, which fully executes steps in topological order and handles tens of thousands of tasks. By design, it lacks high-availability, triggering, and alerting features.

The local scheduler focuses on quick develop-test-debug cycles. Users deploy to a production-grade scheduler like AWS Step Functions when satisfied with results, then continue using both in parallel. After initial deployment, a data scientist typically keeps working locally, possibly deploying an experimental version to production for A/B testing. When production issues arise, the resume command lets users reproduce them locally.

Production Scheduling Requirements

Real-world data science at Netflix-scale places demanding requirements on the scheduler layer:

  • Scale: Graphs can be almost arbitrarily large, especially with dynamic fan-outs via the foreach construct. Some existing workflows train a model for every country (a 200-way foreach) and run hyperparameter searches over 100 parametrizations per model (a 100-way foreach)—producing 20,000 tasks in a single workflow.
  • Concurrency: Hundreds of thousands of active workflows may need scheduling simultaneously, with many variations of the same workflow running at once.
  • High availability: The scheduler must ensure business-critical workflows execute on time, making scalability and availability in the same system a non-trivial engineering challenge.
  • Triggers: Workflows may start on time-based Cron-style scheduling or on external signals. At Netflix, most workflows trigger based on upstream data availability—an ML workflow starts whenever fresh data appears.
  • Observability: A GUI for monitoring execution and alerting mechanisms for critical failures are essential.

The scheduler layer never executes user code. That responsibility belongs to the compute layer. Metaflow provides a local compute layer that runs tasks as local processes using multiple CPU cores. When more resources are needed, both the local scheduler and AWS Step Functions can use AWS Batch to execute tasks as independent containers.

Why AWS Step Functions

Metaflow’s local scheduler covers the development and testing phase, where fast, manual iterations matter more than uptime or unattended execution. Inside Netflix, production workflows are handed to an internal scheduler called Meson. For the open-source community, Metaflow needed a comparable, publicly available alternative that any team could operate.

Popular open-source schedulers such as Luigi and Airflow were evaluated but fell short on the two criteria most important to Netflix: high availability and scalability. AWS Step Functions (SFN) was selected as the first supported production scheduler because it scores well on those requirements and offers several other advantages:

  • High availability without operational overhead. AWS provides a strong SLA, and there is no scheduler cluster to run or maintain. At Netflix, a dedicated senior engineering team operates the internal scheduler; most companies cannot justify that cost.
  • Scalability and generous limits. SFN supports a very large number of concurrent workflows, with individual executions capped at 25,000 state transitions — sufficient for nearly any use case. The maximum execution duration of one year is particularly valuable for long-running machine learning jobs.
  • Event-driven triggers. SFN has built-in mechanisms for launching workflows from external events, which can be composed into a broader web of data and ML workflows over time.
  • Familiar monitoring tooling. Teams already using AWS can rely on CloudWatch for observability and alerting.

The cost of SFN is negligible for most workflows, which fits Metaflow’s philosophy of leveraging the best available managed infrastructure for each layer of the ML stack. A data scientist deploys a flow to production with a single command:

python myflow.py step-functions create

Full usage details are in the Metaflow documentation.

How the translation works

When step-functions create is run, Metaflow performs a static analysis of the workflow defined in the FlowSpec class. The DAG structure is parsed and compiled to Amazon States Language, the native specification format for AWS Step Functions.

The compile step also translates Metaflow Parameters and decorators such as @resources and @retry into equivalent SFN configuration. The user’s code and its dependencies are snapshotted and stored in S3, isolating production runs from any external changes other than input data.

All user-defined code now executes as containers on AWS Batch, the only compute layer currently wired to SFN. Other compute backends may be supported in the future. The beauty of this approach is that the same workflow code runs unchanged on the local scheduler or SFN. Data scientists can iterate rapidly during development, then push to production with a single command — and repeat that cycle as often as needed.

What’s next

The Step Functions integration released today gives every Metaflow user access to a production-grade workflow scheduler similar to what Netflix has run internally for the past three years. Data scientists can learn how to deploy flows from the documentation; infrastructure engineers should consult the Administrator’s Guide to Metaflow.

Step Functions is a strong first choice for scheduling Metaflow in production, but the scheduler layer remains pluggable by design. Teams that have had success with another scheduler or need help getting started with Step Functions are encouraged to reach out.