Airflow at Slack: A Controlled Jump from 1.8 to 1.10
Apache Airflow orchestrates the data warehouse workflows that power Slack’s product and business metrics, as well as internal engineering use cases like search and offline indexing. After running Airflow 1.8 for two years, the team needed to move to 1.10. This upgrade involved a backward-incompatible schema change to the metadata database, which made the migration strategy critical. The primary goals were reliability, fast rollback, minimized downtime, and preserving the history of previous runs to avoid resetting DAG start dates.
Choosing a Migration Strategy
The team considered two possible approaches. A red-black upgrade, running old and new Airflow versions side-by-side, was dismissed early. That approach would require each version to point at a separate metadata database to avoid duplicate task scheduling. Moving DAGs between databases piecewise would inevitably lose historical run data.
The selected strategy was a big-bang upgrade: test in dev, move all DAGs to the new version at once, and then either fix forward or roll back if problems arose. To improve test coverage, critical production DAGs were copied into the dev environment.
Database Rollback Strategy
Two database backup approaches were weighed. The snapshot method was straightforward but slow, given the large size of the metadata database. Restoring from a snapshot would lead to significant catch-up time and overall downtime.
Instead, the team chose a replica-based strategy. At the start, there was one master and one replica. Before the upgrade, two additional replicas were created. When the time came to upgrade the schema, one replica was cut off from replication and promoted to be the new master after running the schema upgrade script. If that upgrade failed, the old master was still available for an immediate rollback.
Handling the Schema Upgrade
The schema changes between Airflow 1.8 and 1.10 are extensive. The official recommendation is to upgrade to 1.9 first, as the 1.10 upgrade command (airflow upgradedb) does not support a direct jump. The team wanted to avoid a two-step process.
They also found the Python-based airflow upgradedb script, which uses Alembic, prohibitively slow on a database with over ten million rows in some tables. The solution was to write a custom MySQL schema upgrade script. Running the equivalent queries directly against MySQL was significantly faster than using the Airflow-provided Python scripts. This custom script was contributed back to the Airflow community (see AIRFLOW-6094).
Issues Found During Testing
The adhoc Attribute Disappeared
Airflow 1.10 removed the adhoc attribute from the task object, which previously prevented a task from being scheduled and marked it as manual-only. DAGs using this feature had those tasks consolidated into a new DAG configured with schedule_interval=None.
Presto Operator Failures
The Presto hook broke due to an incompatibility between the future package version 0.16.0 and the hook when running on Python 2.7. The temporary fix was to convert the protocol field to Unicode. The alternative, migrating Airflow to Python 3, was treated as a large, separate effort.
Risky UI Behavior
In version 1.10.3, the “Mark Success” button on a task would also mark all of its downstream dependencies as successful. This created a risk of accidental data corruption if users misclicked. Slack disabled this feature and instructed users to use the “Task Instance” admin page instead.
Issues Found After the Upgrade
HiveServer2Hook Restricted DESCRIBE
The standard hook stopped allowing DESCRIBE queries, a bug already reported upstream in AIRFLOW-3633. Slack created its own SlackHiveServer2Hook to restore this functionality.
boto2 vs. boto3 Behavior Change
The list_keys API for S3 changed behavior with boto3, causing some tasks to fail. The new implementation ignored S3 folder markers ($folder$ files). The team updated internal library code to handle this new behavior explicitly.
Timezone Implicit Defaults
Airflow 1.10 is timezone-aware. This caused the metadata database to return execution dates in the host’s configured timezone (Pacific time), which broke dashboards expecting UTC. The default timezone in the Airflow metadata DB was updated to UTC to fix this.
Post-Mortem Takeaways
After the upgrade, the team identified several practices that worked well and areas for improvement.
The creation of a detailed runbook with exact, copy-pasteable commands helped prevent mistakes during execution. The upgrade also allowed the team to clear out a significant amount of lingering technical debt. Clear communication with stakeholders in advance of the downtime set proper expectations.
Going forward, the team plans to upgrade Airflow more frequently to avoid large, infrequent jumps. A lack of dev DAGs meant several issues only surfaced in production, so increasing dev coverage is a priority. The need for better internal monitoring is now being addressed with an alerting framework that tracks scheduling timeliness. The bulk clearing of tasks immediately after the upgrade overloaded the EMR cluster; a more disciplined task-restart strategy is needed for future migrations. Finally, an internal code freeze during major upgrades prevents unrelated changes from introducing new breakages.
