The People Behind Slack's 100-Petabyte Data Platform

Slack's Data Engineering team owns the company's data lake, analytics dashboards, and the services that sit between them. That lake has grown from under a petabyte to over 100 petabytes in recent years, spanning millions of tables. The engineers profiled here work across orchestration, metrics foundations, infrastructure, and governance—and they represent a deliberate push to keep the team's perspectives as diverse as its data.

Orchestration and Migration Work

Jessica Stewart, a Senior Software Engineer on the Data Orchestration team, joined in May 2023. Her team manages internal data workflows built on Apache Airflow and Apache Pinot within a primary cluster storing terabytes of data. The system maintains sub-second query latency and a roughly 99.95% query success rate SLA, supporting both internal employee tools and Slack's user-facing analytics dashboards.

A current project involves moving from virtual machines to a cloud-native Kubernetes infrastructure. VMs carried higher infrastructure costs and maintenance overhead, so the team was eager to containerize. Slack's internal Kubernetes platform doesn't fully support Helm charts or Kubernetes services, so the team made customizations to the open-source setup, including a custom networking configuration to integrate Pinot with the internal platform. They've also built custom Python tooling that wraps around Pinot to standardize operations, streamlined data ingestion through Airflow pipelines, and automated deployments using Ansible, Kubernetes, and Jenkins.

Standardizing the Compute Stack

Nilanjana Mukherjee, a Staff Software Engineer on the Metrics Foundations team, joined in October 2021. Her team generates actionable datasets and maintains high data accuracy with landing time SLAs. She led a company-wide migration of workloads from Hive to Spark 3, becoming one of Slack's earliest Spark adopters. That made her the Spark subject matter expert for over 40 teams across the organization. She was recently promoted from Senior Engineer to Staff Engineer.

Ramya Sundaresan, a Senior Software Engineer also on the Metrics Foundations team, joined in May 2022. She has worked with both the Data Ingestion and Metrics Foundations teams. Her contributions include implementing Apache Iceberg within Kafka Connect clusters, orchestrating the migration of Airflow from AWS EC2 instances to Kubernetes, and migrating jobs to Spark 3 on AWS EMR 6 clusters. The shift from Spark 2 on EMR 5 to Spark 3 on EMR 6 aligned with a company-wide objective to unify the tech stack and reduce reliance on legacy systems like Hive and MapReduce. Ramya led multiple teams through the transition using parallel build pipelines and comprehensive documentation, achieving the migration goal in under a year.

Modernizing Ingestion Infrastructure

Shrushti Patel, a Senior Software Engineer on the Data Infrastructure team, joined in August 2020. Her team owns the infrastructure and services for delivering reliable, timely data, working with AWS EMR, Airflow, Trino, Secor, and Ranger. When she joined, Slack relied on Secor for transferring Kafka data to S3 on EC2—a setup that struggled when incorporating new data topics. Secor lacked industry-standard recognition, ongoing development, and support for emerging data formats.

Patel took over the Secor setup and led its migration to Bedrock, an internally-developed Kubernetes framework. That simplified adding new topics and produced cost savings through resource optimization. The team is now migrating from Secor to Kafka Connect, which offers out-of-the-box state management, fault tolerance, and scaling by operating as an abstraction layer that leverages connectors as executable JARs. The shift moves the platform toward real-time streaming.

Balancing Big Projects and New Parenthood

Nathalie Kaligirwa, a Senior Software Engineer on the Metrics Platform & Governance team, joined in November 2021. Her team builds scalable, standardized tools for the data experience. Over the past two years she took six months of maternity leave while working on complex, multi-stakeholder projects. Metrics standardization is a major initiative at Slack, encompassing projects like Merlin, a framework for abstracting metrics and wide-table creation. During parenthood, Kaligirwa focused on manageable components within that initiative, such as updating a service to dynamically add new Merlin-created metrics, eliminating manual migrations. She then shifted to improving the search experience, laying groundwork for a new metadata service that provides lineage across multiple data tools and scalable search.

Leadership's Role in Retention and Growth

Slack's Data Engineering leadership—represented by Suzanna, Lakshmi, and Beate—describes a culture built on mentorship, mobility, and support for significant life events. The team encourages internal transfers for professional growth and skill diversification. They also point to a deliberate structure around maternity leave: reintegration is organized so returning teammates get context on ongoing projects without friction.

Leadership's approach to supporting women in data engineering centers on four practices:

  • Visibility: Amplifying women's voices by encouraging leadership roles, conference speaking, and participation in industry events.
  • Mentorship: Offering formal and informal programs connecting experienced women leaders with mentees for guidance and support.
  • Advocacy: Actively challenging biases and advocating for fair practices across the work environment.
  • Empowerment: Supporting flexible work arrangements, diversity and inclusion initiatives, and access to training resources.

These efforts show in the growing number of women joining the teams and in the experiences described by the engineers above. For a team managing infrastructure at 100-petabyte scale, that diversity of perspective isn't just a cultural value—it's part of how the team keeps solving increasingly complex data problems.