Why Slack moved from Spark 2/Hive 2 to Spark 3

Slack Data Engineering runs a federated AWS EMR cluster model to support data warehousing across over 40 internal teams. The data warehouse stores physical data in S3, with metadata managed by Hive Metastore on RDS. SQL handles most workloads, while Scala and PySpark cover more complex cases. Airflow orchestrates thousands of DAGs through custom Spark operators that submit jobs to the EMR clusters via Livy's Batches API.

As data volumes grew, the team frequently missed critical pipeline landing-time SLAs. The search for an alternative to the existing EMR 5 setup—which ran Spark 2.4.8 and Hive 2.3.9—pointed to Spark 3 on EMR 6. The core motivators were performance improvements from Adaptive Query Execution (AQE), updated log4j libraries for security, and potential cost savings. The migration plan split into two phases: first upgrading EMR from 5.x to 6.x, then moving the compute engine from Hive 2.x/Spark 2.x to Spark 3.x.

Handling a dual-version transition

Given the scale—over 60 EMR clusters and thousands of Airflow DAGs—a gradual migration was necessary. The team decided to run EMR 5 and EMR 6 side by side until workloads could be moved without disrupting existing product roadmaps. This dual-mode operation raised several infrastructure questions, each addressed with a specific solution.

Keeping a single Hive catalog

Both Spark 2 and Spark 3 workloads needed to use the same Hive catalog while the migration ran over multiple quarters. The solution was to upgrade the existing Hive Metastore schema from 2.3.0 to 3.1.0 using the Hive Schema Tool, executed on the EMR 5 master host that connected to the catalog database. Before the upgrade, the team backed up the Hive Metastore database and scheduled downtime for job processing. The upgraded HMS 3 catalog remained backward compatible with Hive 2 and Spark 2 applications, so both EMR 5 and EMR 6 clusters could use the same catalog post-upgrade.

Provisioning mixed EMR versions

EMR clusters are launched through the AWS SDK RunJobFlow API, which takes a JSON launch configuration. The team maintains a base config and overrides fields like InstanceFleets, Capacity, and Release Label per cluster. For EMR 6, new configs were created with autoscaling enabled and a low minimum capacity to control cost. During the migration, the team gradually shrank the EMR 5 fleet and grew the EMR 6 fleet based on actual usage.

Building JARs for both Spark versions

Code is built with Bazel. To support both engines at once, the team implemented parallel build streams for Spark JARs across versions 2.x and 3.x, propagating all config changes to both JARs. A --config=spark3 flag in the .bazelrc file enables building the Spark 3 version locally for testing. In Airflow pipelines, the operator picks the correct JAR automatically as jobs are migrated.

Routing jobs with an operator flag

The custom Airflow Spark operator was enhanced with a boolean flag that routes jobs to either pre- or post-migration clusters by a simple toggle. The operator also gained four logical sizing groups—SMALL, DEFAULT, LARGE, and EXTRA_LARGE—each with its own driver memory, executor memory, and executor ranges. These presets let users migrate existing Hive jobs to Spark 3 without deeply understanding Spark configuration knobs.

Adapting code for Spark 3

Most existing Hive and Spark 2 code ran as-is on Spark 3, but a few patterns required changes. Skew handling was one notable area. Some jobs used bulky subqueries to generate salt keys, while others used RAND() in the join key, a workaround that works in Hive but fails in Spark 3 with:

org.apache.spark.sql.AnalysisException: nondeterministic expressions are only allowed in Project, Filter, Aggregate, or Window.

The team removed this custom skew-handling code entirely, letting AQE manage skewed joins instead. Spark 3 also enforced stricter data type casting rules. For compatibility, the team changed the default of spark.sql.storeAssignmentPolicy from ‘ANSI’ to ‘Legacy’. A separate issue surfaced when Spark 3 could not reconcile schemas between Hive Metastore and underlying Parquet files, causing java.lang.StackOverflowError. Setting spark.sql.hive.convertMetastoreParquet to False resolved it.

Validating migration output

Exact data match validation was required, not sampling, because some datasets—like customer billing data—are mission-critical. The team compared a production table from EMR 5 (prod_table_hive2_or_spark2) against a test table created on EMR 6 (test_table_spark3). Config files and macros let SQL scripts read from the production schema and write to a test schema, populating the test table using Spark 3. Validation ran through Trino with except and count queries for speed. When mismatches appeared, an in-house Python framework with the Trino engine was used for deeper analysis. Production runtime of pipelines was monitored continuously using Airflow metadata database tables.

Several patterns created false positives in validation:

  • Code relying on the current timestamp produced variations between runs, so timestamp columns were excluded from comparisons.
  • Missing a differentiable order by clause caused random row ordering where ties existed; code was fixed to include a proper sort key.
  • Built-in functions behaved differently across engines. For example, Hive and Spark differ on how Greatest handles a NULL argument, so code was adjusted to match the intended business logic.

Measured wins after the move

Runtime comparisons drawn from the Airflow metadata DB (the duration column in task_instance) showed broad gains. Most Airflow tasks landed 30–60% faster, and select jobs saw up to 90% runtime improvement. The chart below illustrates one critical task’s post-migration runtime.

S3-optimized committer fixes

The EMRFS S3-optimized committer on EMR 6 resolved two recurring headaches for Spark jobs handling text-based input/output: incomplete writes and misleading SUCCESS statuses. It also cut overhead by removing list and rename operations during job and task commit phases. This was previously available only for Parquet; as of EMR 6.4.0, support extends to ORC and text formats including CSV and JSON.

Adaptive Query Execution in practice

Query plans confirmed AQE doing the heavy lifting. Dynamic skew join optimization let us delete several lines of hand-rolled skew handling and replace them with a plain join on the key—no extra logic required. The following plan snippet shows the AQE (skew=true) hint at work.

Shuffle partition coalescing was just as useful. Setting a sufficiently large spark.sql.adaptive.coalescePartitions.initialPartitionNum lets Spark pick the right partition count at runtime; one plan shows partitions dropping from 3000 to 348 without any manual tuning.

Bottom line

The EMR 6 upgrade delivered material gains in runtime, efficiency, and reliability across our data pipelines. AQE reduced manual tuning and removed bespoke skew code, while the S3-optimized committer stamped out write-corruption and false-status incidents. The migration itself ran without a single incident at any step, and we cleaned up the pipeline codebase along the way, onboarding engineers onto a modern Spark 3 foundation. That also opens the door to newer lakehouse formats like Iceberg and Hudi on EMR 6. Long-term modernization efforts like this are worth the investment—the efficiency returns show across the board.