Smoothing Airflow's Rough Edges at Scale

Apache Airflow has been in production at Shopify for more than two years, handling everything from data extractions and machine learning model training to Iceberg table maintenance and DBT-powered data modeling. The deployment currently runs Airflow 2.2 on Kubernetes with the Celery executor and MySQL 8.

Growth has been aggressive. The largest environment alone runs over 10,000 DAGs, averages more than 400 concurrently executing tasks, and processes upwards of 150,000 runs per day. That scale exposed several weaknesses in how we originally deployed Airflow, and the fixes we built along the way are worth sharing.

File Access Bottlenecks

Airflow's scheduler keeps its internal model of workflows in sync by repeatedly scanning and reparsing every file in the DAG directory. Those files need to be consistent across all schedulers and workers in an environment, and the scanning has to happen often. Slow file access translates directly into stale jobs and sluggish scheduling.

Our first attempt at keeping DAG files consistent used GCSFuse to mount Google Cloud Storage into every pod. That worked initially, but at scale it became a bottleneck: every file read triggered a request to GCS, and every pod mounted the bucket independently, multiplying the volume of requests.

The fix was to run an NFS server inside the Kubernetes cluster and mount it as a read-write-many volume into worker and scheduler pods. A custom script in a separate pod synchronizes the NFS volume with GCS, so users still interact with GCS for uploading DAGs. The sync script can conditionally pull only a subset of DAGs from a given bucket, or combine DAGs from multiple buckets into one file system based on the target environment's configuration.

This arrangement gives us fast local file access while keeping GCS as the durable source of truth. It also lets us lean on GCP's IAM for access control: users can upload directly to staging, but production uploads are restricted to continuous deployment processes.

Beyond storage, Airflow's file processing itself is tunable. Options like the parsing sort mode, parsing process parallelism, and the DAG file processor timeout let you balance interactive DAG development against scheduler throughput.

Metadata Growth Slows Everything Down

In a typical Airflow installation, metadata accumulation isn't a problem for years. At our scale, it becomes one quickly. The database gets heavier, the Web UI slows down, and upgrades become painful as migrations stretch into hours.

We settled on a 28-day metadata retention policy. A simple DAG using PythonOperator and ORM queries deletes rows from tables holding historical data—DagRuns, TaskInstances, Logs, TaskRetries, and similar. Twenty-eight days gives us enough history for incident management and performance tracking while keeping the database lean.

The tradeoff is that features relying on durable job history, like long-running backfills, aren't viable in our environment. That hasn't hurt us, but it's worth weighing against your own retention needs. For those who'd rather not build a custom cleanup DAG, Airflow 2.3 added a db clean command that removes old metadata natively.

Knowing Who Owns What

In a multi-tenant Airflow environment, you need to trace any DAG back to a person or team quickly—especially when a job is failing or interfering with others. If every DAG came from a single repository, git blame would suffice. But we let users deploy from their own projects and even generate jobs dynamically at deploy time, which makes ownership murky.

Our solution is a per-environment manifest file: a YAML configuration where users must register a namespace for their DAGs. The manifest records the owning team, the source GitHub repository or GCS bucket, and basic restrictions for jobs in that namespace. We maintain a separate manifest per environment, stored in GCS alongside the DAGs themselves.

Guardrails for DAG Authors

Allowing users to write and upload DAGs directly means granting access to a central piece of the data platform with wide-reaching connections. We trust our users, but at scale it's impossible for administrators to review every job before it reaches production. So we built guardrails into a DAG policy that reads the manifest and rejects non-conforming DAGs by raising AirflowClusterPolicyViolation.

The policy enforces several constraints based on the manifest contents:

  • DAG IDs must be prefixed with a registered namespace, establishing ownership.
  • Tasks may only enqueue to the celery queue designated for their namespace.
  • Tasks may only use pools allocated to their namespace, preventing capacity theft.
  • KubernetesPodOperator tasks may only launch pods into approved namespaces and external Kubernetes clusters, limiting access to other teams' secrets.

The policy is extensible: you could restrict operators to an allowlist or mutate tasks to conform to a specification, such as adding a namespace-specific execution timeout to every task.

The Surge Problem in Scheduling

Absolute intervals like timedelta(hours=1) are convenient, but they create rhythmic surges. When a large set of automatically generated DAGs lands—or a single Python file generates many DAGs at parse time—all of their runs are created simultaneously. This overloads the scheduler and whatever external systems the jobs depend on, then repeats itself every interval.

Crontab schedules don't have that problem, but they introduce a different one: humans gravitate to the top of the hour, midnight, and other tidy round numbers. Sometimes that's genuinely required, but often it's just habit, and it creates uneven load on external services.

For automatically generated DAGs—the vast majority of our workflows—we use deterministic, randomized schedule intervals derived from a hash of a constant seed like the dag_id. This smooths out the load considerably, as task completion rates across a twelve-hour window in our largest environment show. The main limitation is that not every interval can be expressed as a single crontab line. We haven't found that restrictive in practice; when we absolutely need a five-hour interval, we accept one four-hour gap per day.

Contention Is Everywhere

Resource contention in Airflow isn't confined to one layer. It's easy to chase bottlenecks through configuration changes only to find the next one downstream. Some conflicts can be resolved inside Airflow, but others require infrastructure changes. The two biggest levers we've found are being deliberate about schedule distributions—as described above—and using the policy layer to enforce pool and queue separation between namespaces, so one team's workload can't starve another's. Beyond that, it's about knowing your deployment's fault lines and tuning each layer independently rather than treating contention as a single problem to solve.

Managing Resource Contention

Airflow provides several mechanisms for controlling how tasks compete for resources. Pools limit concurrency for a defined set of tasks, which helps smooth out traffic bursts that would otherwise create disruptive spikes. The trade-off is operational: only administrators can edit pools through the Web UI, which makes them awkward to maintain as infrastructure.

Shopify solved this with a custom DAG that synchronizes pool definitions in the environment against a Kubernetes Configmap via simple ORM queries. The effect is that pools become part of the deployment configuration and can be changed through a reviewed pull request, without granting users administrator access to the Airflow instance.

Prioritizing Critical Tasks

For tasks that need to run promptly, priority_weight controls ordering within the scheduler queue. A task with a high weight is scheduled before lower-weighted tasks. Shopify applies this to its basic Airflow monitoring DAG, which emits metrics and powers alerts, to ensure that these lightweight but critical tasks execute without delay.

The default weighting rule complicates things: the effective priority of a task is the sum of its own weight plus the weights of all downstream tasks. This means that upstream tasks in large DAGs get a natural boost over tasks in smaller DAGs. Setting a meaningful priority_weight therefore requires knowing how other DAGs in the environment are structured.

Isolating Workloads with Celery Queues

For workloads that must run in separate execution environments—different Python libraries, higher resource allowances, or different access levels—Airflow supports multiple Celery queues. Workers can be configured to pull from specific queues, and individual tasks can be assigned to a queue via the queue operator argument. Starting a worker for a different queue is a simple command:

bashAirflow celery worker –queues <list of queues>

Isolating workloads this way protects latency-sensitive or high-priority jobs from contending with other workloads for worker resources.

Pools, priority_weight, and queues are complementary. Pools constrain concurrency inside a workload; priority_weight gives specific tasks low-latency access to the scheduler; queue-based isolation provides granular control over a task's execution environment. However, not every finite resource can be rationed in airflow. Scheduler throughput, database capacity, and Kubernetes IP space are limited globally, and without creating fully isolated environments, they cannot be apportioned per workload.

Key Takeaways

  • A mix of GCS and NFS balances performance with ease of file management.
  • Enforcing metadata retention policies prevents long-term performance degradation.
  • A centralized metadata platform can track DAG origins and owners.
  • DAG Policies enforce standards and structural limits on jobs.
  • Standardized schedule generation reduces traffic bursts.
  • Airflow's multiple scheduling mechanisms allow for fine-grained resource management.

Shopify's next step is applying these lessons in a multi-environment setup. Splitting workloads across separate Airflow instances should make the platform more resilient, allow each instance to be tailored to the needs of its workloads, and reduce the blast radius of any single deployment.