Why Python 3 migration required a different playbook

Slack’s previous Airflow upgrade, from 1.8 to 1.10, was a forced “big bang” operation. When Python 2 reached end of life, the team faced a second major migration — this time of the runtime itself. Rather than repeat the earlier approach, the engineering team designed a red-black deployment that ran Python 2 and Python 3 Airflow infrastructure side by side, then shifted DAG execution over gradually.

Apache Airflow is Slack’s workflow scheduler for data warehouse jobs, covering product metrics, business metrics, search, and offline indexing. The environment runs hundreds of workflows and tens of thousands of tasks daily, so any migration had to be invisible to DAG authors and users.

The migration followed seven phases: build a Python 3 virtual environment, launch Python 3 Celery workers, clean up stale DAGs, fix Python 3 incompatibilities, move DAGs to the new workers, switch the Airflow services, and remove the Python 2 infrastructure.

Order matters: the scheduler and webserver must stay on Python 2 until DAGs are compatible. A Python 3 scheduler cannot parse Python 2-only DAGs, so switching services early would break the entire system.

Building the Python 3 environment

Slack’s Airflow boxes depended on system Python 3.5.2, which is too old for recent Airflow versions. The team installed pyenv with its virtualenv plugin to decouple Python from the OS and managed both Python 2 and 3 dependencies through Poetry, defining everything in a single pyproject.toml.

Virtual environments were built, shipped to Airflow boxes, and made relocatable. Python 3 deprecated virtualenv’s --relocatable flag, so the team instead replaced shebang lines in environment files with the final install location:

Standing up Python 3 workers

A new fleet of Celery workers ran the Python 3 virtual environment alongside the existing Python 2 workers. The workers listened on python2 and python3 queues in Redis; queue creation is automatic when the Airflow scheduler submits a task with a queue configured.

One infrastructure fix was required before the two worker fleets could coexist. Airflow stores some metadata in serialized form, and SQLAlchemy writes Python 3 pickles using a higher protocol version than Python 2 can read. This caused Celery failure states to not propagate to the Python 2 scheduler. The team patched SQLAlchemy so Python 3 workers serialize with protocol version 2, keeping the metadata readable by both runtimes. The patch lives in SQLAlchemy’s sqltypes.py and can be removed once the migration is complete.

Making DAGs run on both runtimes

DAGs needed to be Python 2/3 compatible, not just Python 3 compatible. The scheduler still ran on Python 2 and had to parse every DAG, and keeping compatibility made rollback a queue-config change rather than a code change.

Rather than editing files by hand, Slack evaluated two automatic converters: 2to3 and futurize. Futurize won because it produces code that runs on both Python 2 and 3, while 2to3 is designed for one-way porting.

Futurize’s fixers are split into stage 1 (safe) and stage 2 (potentially unsafe). The team applied them individually, testing and deploying over several days so problems could be isolated quickly:

To prevent regressions, a CI test was added to PRs that detects Python 3 incompatibilities of the types already fixed.

Migrating DAGs team by team

Moving hundreds of DAGs required an organizational approach. Slack migrated them team by team: each team provided a point of contact, DAGs were prioritized by criticality, and the least critical were moved first. DAGs were tested locally before the switch.

Runtime issues were fixed forward, with rollbacks rarely needed. After enough DAGs had moved successfully, the remaining ones were switched en masse.

Instead of editing the queue attribute on each DAG, the team kept a file listing all migrated DAGs. Code read that file and set the configuration automatically, so adding or rolling back DAGs required touching just one file.

Final service switch and cleanup

Once all DAGs ran on Python 3 workers, the scheduler, webserver, and Flower were switched to the Python 3 virtual environment. Python 2 references were removed, and the old worker fleet was terminated.

Common issues and their fixes

  • TypeError: a bytes-like object is required, not 'str' — the most frequent error, caused by Python 3 treating strings as Unicode. Fix: explicitly call encode() to convert strings to bytes.
  • Snowflake connector failures — errors like asn1crypto.keys.PublicKeyInfo().unwrap() has been removed appeared with Snowflake operators. Fix: upgrade snowflake-connector-python to 2.x.
  • Dictionary key ordering — Python 3 does not guarantee predictable key order. Fix: use OrderedDict to preserve insertion order.
  • Broken DAG with UTF-8 decode errors — file reads defaulted to ASCII in Python 3. Fix: specify encoding explicitly when opening files.
  • Embedded Python in SQL files missed by futurize — futurize only scans Python files. Fix: manually convert the small set of SQL scripts containing Python code.
  • OpenSSL issues on Mac OS Catalina — running Airflow locally produced Abort trap: 6. Fix: run specific OpenSSL-related commands to resolve the environment issue.

Outcome

The red-black deployment was transparent to users and recorded zero SLA misses. Running both Python 2 and 3 infrastructure concurrently let Slack validate changes incrementally and roll back without code changes. With the migration complete, the team can use Python 3 features and is positioned for an Airflow 2.0 upgrade.