Data Engineering Interns Tackle Two High-Impact Projects at Slack

Slack’s Data Engineering internship program places software engineers on the company’s core data infrastructure, working alongside full-time engineers on production systems. Interns were onboarded through a structured two-week process: the first week focused on Salesforce and Slack integration through meetings, breakout sessions, and hands-on tutorials; the second was dedicated to the Data Engineering pillar, covering setup documentation and team-building to ensure new hires felt connected from day one.

Interns were divided across two major initiatives spanning four teams, each addressing a real operational pain point within Slack’s data platform.

Improving Data Search and Discovery

The first project, the Reliable Data Discovery Tool, was born from a scalability problem. Slack’s internal analytics platform allows employees to search across roughly 1,700 dashboards, 1.8 million queries, and 11,000 tables. As those numbers grow, search becomes slower and less reliable, and users lack signals about which data assets are trustworthy or relevant to their use case.

The solution had two parts. On the backend, the team used OpenSearch—an open-source search and analytics platform built on Elasticsearch—to index and search metadata from the Data Engineering team’s internal data assets, returning ranked results and displaying reliability metadata alongside each hit to address scalability concerns.

On the frontend, the intern team used TypeScript, React, and AntDesign to build a flexible search UI where users can find, sort, and filter data. The interface surfaces information like title, owner, monthly active users, and warnings, and it stores search state in the URL to make sharing findings easier. Integration work between frontend and backend was handled via Fast API, giving the interns experience bridging the two layers.

Tracking Job Performance to Prevent Failures

The second project addressed the thousands of data pipelines Slack runs daily, which generate petabytes of data. When processing jobs fail due to performance issues, engineers must investigate manually—a reactive process that consumes time that could go toward other work.

The team’s solution pulls data from two sources: the Spark History Server, which provides performance metrics such as memory usage and runtime, and Apache Airflow, Slack’s job scheduler, which provides data on team ownership and performance expectations. The interns set up ingestion jobs to query the Spark server’s API and parse the performance data, then built pipelines to pull Airflow fields with validation checks.

The two datasets were merged into an enriched table, combining job performance with team identification and configuration details. From that table, the team created dashboards showing the most computationally expensive tasks and historical job trends, along with alerts that signal when a job might fail. This end-to-end view of job history didn’t exist before the project, giving teams a proactive way to optimize resource usage and reduce failures.

A Program Management View

A Technical Product Management intern worked cross-functionally on the same initiatives, in addition to managing two concurrent flagship programs: Data Retention, tied to Slack’s privacy goals, and Legacy Users Deprecation, which replaced older tables with faster, modular ones.

The role involved aligning product and engineering vision, communicating with stakeholders, and surfacing risks and dependencies. The TPM intern highlighted several takeaways:

  • Identifying and aligning on scope early allowed epics to be broken into actionable stories while mitigating risks upfront.
  • Consistent stakeholder management through meetings, documentation, and weekly updates enabled seamless collaboration.
  • Flexibility was essential, as requirements evolved rapidly and agile principles helped when plans changed.
  • Building foundational data and systems knowledge made it easier to contextualize projects and communicate with technical partners, and also enabled automating manual processes.
  • Soft skills like patience and empathy were key to working through technical challenges and building strong team connections.

What Made the Internship Successful

Across both projects, the interns credited several factors for their success. Working on real-world projects with cross-functional teams gave them hands-on experience beyond textbook knowledge, with mentors and managers providing guidance to help them exceed what they could have accomplished alone. The welcoming Slack culture—fostered through activities like office onsites and lunch outings—built camaraderie that improved collaboration. Navigating challenges, from relocating to new cities to learning unfamiliar data systems, strengthened their resilience and adaptability, laying a foundation for future careers in data engineering.