Rethinking a Decade of SSH Dependencies

Slack’s data platform grew around a 2017 pattern: Airflow orchestrated jobs on AWS EMR clusters by SSHing into master nodes. What started as a direct path became the backbone of the platform. By 2024, more than 700 production jobs—including daily Solr search indexing over terabytes of data—depended on SSH-based operators.

That architecture carried real weight. Every job meant direct, stateful connections into production clusters. Key rotation and permission management across orchestration workers was operationally heavy. Audit trails required correlating logs across multiple systems. And the model had reliability gaps: Kubernetes pod restarts killed SSH sessions, long-running jobs could outlive their connections as orphans, and there was no trustworthy way to confirm success after a disconnect.

Those problems weren't just annoying—they were blocking. Slack couldn't move Spark workloads onto Kubernetes or EKS, couldn't finish migrating the last main-account EMR clusters to child accounts under the Whitecastle initiative, and couldn't build proper job monitoring. The SSH layer had to go entirely.

The REST Pattern That Replaces Connections

The alternative to SSH is not a new tool. Modern compute engines (YARN, Trino, Snowflake) already expose HTTP APIs for job submission. The lifecycle is fundamentally different:

  1. POST a job request and receive a job ID
  2. GET job status using that ID
  3. DELETE the job for clean cancellation

Jobs are managed server-side. A client can crash and restart while the job continues running, and status remains queryable. No hanging up mid-phone-call and hoping the other person finishes the conversation.

The catch was coverage. YARN's REST API handles Hadoop workloads well, but what about the 300+ CLI-based jobs running arbitrary shell commands like aws s3 sync or hadoop distcp? Spark and Hive had Livy and HiveServer2 respectively, but MapReduce and shell commands had no off-the-shelf REST path.

YARN Distributed Shell: The Missing Piece

The team evaluated building a custom wrapper service, adopting remote execution frameworks like Ansible or Salt, or creating a new YARN job type. All of those options meant custom security layers, new dependencies, or significant maintenance burdens.

Then they found YARN's Distributed Shell (org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster). It allows any shell script to run in a proper YARN container with resource allocation and lifecycle management—and it was already part of YARN, used the same REST APIs, and required no additional security infrastructure.

The flow works like this:

  1. Upload a command script to S3 (e.g., command.sh in s3://bucket/)
  2. Submit a job to YARN with Distributed Shell configuration
  3. YARN allocates a container, downloads the script, and executes it

YARN then manages resource limits (memory, vCores), container isolation, retries and fault tolerance, clean cancellation, and logging through the YARN UI.

YARN Distributed Shell job submission flow

That single architectural decision unlocked the entire migration. Not just Hadoop workloads—any shell command could now run in a managed YARN container.

Quarry: The Universal Job Submission Gateway

Slack already had Quarry, a REST-based job submission gateway originally built as a unified interface across compute engines like EMR/YARN, Trino, and Snowflake. It handled authentication via service-to-service tokens instead of SSH keys, submitted jobs via REST, tracked state server-side, managed job lifecycle through cancellation and cleanup APIs, and provided structured logs, metrics, and tracing.

Adding YARN Distributed Shell support turned Quarry into the single path for all job types—Spark, Hive queries, or plain shell scripts. The architectural shift was straightforward:

Before: Airflow operators maintained direct SSH connections to EMR master nodes.

Airflow → SSH Connection → EMR Master Node → Execute Command

After: Airflow operators make HTTP requests to Quarry, which submits jobs to YARN and polls for status. If an Airflow pod restarts, the job keeps running and Quarry maintains reliability.

Airflow → Quarry REST API → YARN ResourceManager → EMR Container

Architecture comparison showing the shift from SSH-based direct execution to REST-based job

Migration in Phases, Without Downtime

The migration spanned 700+ jobs across 8 independent data regions, each with distinct network configurations and data sovereignty requirements. The project moved in five deliberate phases:

  • Phase 1 – Proof of Concept: Pilot jobs validated the Quarry-based approach in dev environments.
  • Phase 2 – Security Review: Security teams planned credential elimination and confirmed the REST architecture met requirements.
  • Phase 3 – OKR-Driven Execution: Migration became a Key Result with executive visibility, keeping it prioritized. This phase hit the 80% milestone.
  • Phase 4 – Bulk Migration: Multiple teams (Search Infrastructure, Data Engineering & Analytics, ML Services) coordinated to migrate remaining workloads across all regions in parallel.
  • Phase 5 – Final Cleanup: Overlooked DAGs were completed and all legacy SSH-based operators were deprecated.

The numbers tell the story: 700+ jobs migrated across 7 operator types, 8 regions with coordinated rollouts, 5 teams transitioned, zero downtime for critical services, and 100% SSH elimination achieved in three quarters from the initial pilot.

Uncovering Hidden Dependencies

No migration of this size passes without friction. The move from SSH to Quarry’s REST-based submission surfaced several classes of problems that had been quietly masked by the old approach.

Virtual Memory Check Failures

Jobs that ran flawlessly over SSH began failing with virtual memory (vmem) check errors once migrated to a data export DAG. The root cause: SSH commands executed directly on the master node, bypassing YARN’s resource enforcement entirely. Quarry submits jobs through YARN, which enforces limits the SSH path never applied.

Following AWS best practices, we disabled vmem checks cluster-wide:

"yarn.nodemanager.vmem-check-enabled": "false"

AWS recommends this because Linux virtual memory accounting can be unreliable; physical memory limits provide sufficient enforcement. The broader lesson: migrating from SSH to proper YARN submission will surface resource limits that were previously invisible. Test in dev environments before any production rollout.

Network Segregation and EKM Connectivity

While moving dev search infrastructure jobs from one dev cluster to a staging analytics cluster, a task hit an Enterprise Key Management (EKM) connectivity timeout.

Error: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException:

Unable to execute HTTP request: Connect to sts.amazonaws.com:443 failed: connect timed out

The original cluster had network routing to the necessary key management endpoints. The staging analytics cluster sat in a stricter network segment without that connectivity — a hidden dependency on network topology that job configuration never captured. The fix involved relocating search infrastructure tasks to a dev ETL cluster with proper routing to dev services, while tasks needing production Hive catalogs stayed in staging. We scaled up the dev ETL cluster to absorb the extra load.

The lesson: know your network segregation and account boundaries before deciding where jobs run. Dev jobs need dev network access; prod jobs need prod access.

Multi-Region Complexity

Slack runs EMR clusters across 8 independent data regions for data sovereignty. The SSH deprecation was effectively 8 parallel migrations, each with its own failure modes:

  • Configuration management: Each region needed separate Quarry configurations, cluster endpoints, and routing rules.
  • Testing overhead: Changes required validation across all 8 regions before production, multiplying testing effort.
  • Staggered deployments: Rollouts had to proceed region by region, not simultaneously.
  • Region-specific issues: Network setups, sovereignty rules, and cluster versions varied.

We validated changes in a single pilot region (typically US-based for iteration speed), documented regional requirements, and built region-aware Quarry operators. Rollout to remaining regions proceeded incrementally, learning from each deployment while tracking progress separately per region.

Results After Full Migration

Production now runs 100% through Quarry with REST-based submission; SSH access is eliminated entirely.

Security Gains

Removing SSH access across all 8 data regions substantially reduced the attack surface. SSH key distribution was replaced with service-to-service token authentication. REST API logging provides a proper audit trail — every job submission has structured logs through Quarry, eliminating "who ran that command?" investigations.

This also completed the Whitecastle initiative by enabling migration of the last AWS main account EMR cluster to a child account. Compliance simplified without special security group configurations or complex SSH permission management.

Operational Improvements

Master node resource contention is gone. Non-Hadoop jobs now run in distributed YARN containers with proper resource allocation instead of competing on the master node. Job reliability improved dramatically: jobs survive client Kubernetes pod restarts because Quarry tracks jobs server-side. No more zombie processes, and cancellation through REST APIs delivers clean lifecycle management.

Observability transformed. Structured job status, logs, and metrics are available through Quarry’s API, enabling full lifecycle tracking and debugging with proper tooling rather than SSH guesswork.

Future Enablement

The REST architecture unblocked several initiatives:

  • Spark on Kubernetes migration possible with no SSH dependencies to convert.
  • Cloud-native architecture patterns enabled.
  • Simpler onboarding via maintainable Quarry operators vs. complex SSH configurations.
  • Platform evolution by decoupling Airflow from EMR infrastructure.
  • Standardized submission consolidated through Quarry for easier future changes.

Two years of production experience confirm the decisions: the REST-based approach delivered on security, stability, and flexibility.

Lessons for Large-Scale Migrations

What Worked

  1. Incremental migration: Moving Dev → GovDev/CommDev → Prod minimized risk. Migrating by operator type rather than all at once let each batch refine the approach.
  2. Team collaboration: Multiple teams across search, analytics, data engineering, ML, and marketing coordinated. Prompt code reviews and shared-channel communication kept momentum.
  3. Analytics-driven tracking: An analytics dashboard queried the Airflow database to identify remaining SSH-based tasks, making migration progress visible and data-driven.

What We’d Do Differently

  1. Earlier network topology mapping: Network segregation issues like EKM connectivity surfaced late. Document account boundaries and routing before cluster migrations start.
  2. Earlier resource limit testing: The vmem issue caught us late. Test YARN resource limits against an SSH baseline during the pilot phase — SSH bypasses much that will bite in production.
  3. Better restriction communication: When SSHOperator was restricted to prevent new usage, some teams were unaware. Advanced notice to all Airflow users prevents friction.

Best Practices

  1. Monitoring before migration: Build tracking dashboards early using Airflow database queries to identify remaining work and keep progress visible.
  2. Multi-environment testing: Dev, CommDev, and GovDev caught environment-specific issues before production. Network segregation problems appear only across account boundaries.
  3. Progressive operator deprecation: Deprecating operators one at a time (CrunchExecOperator, then S3SyncOperator, etc.) slowed the process but greatly reduced risk.