The Problem: An Expensive Assumption

Slack's Developer Experience (DevXP) team maintains a CI/CD pipeline for one of its largest monorepos, running end-to-end (E2E) tests against the entire stack—frontend, backend, database, and the services in between—before any merge into main. The pipeline was reliable but slow, averaging about 10 minutes per run. The bottleneck wasn't the 200+ E2E tests, which took roughly 5 minutes. The real cost was the frontend build step, which also consumed 5 minutes, even when a pull request (PR) contained no frontend changes whatsoever.

With hundreds of PRs merged daily, the math was stark. The team was triggering thousands of frontend builds per week, each storing nearly a gigabyte of artifacts in AWS S3. Roughly half of those builds were identical to the last merge into main from a frontend perspective, creating terabytes of redundant data and thousands of hours of unnecessary pipeline wait time every week.

Stages of the end to end testing pipeline.

The Fix: Reuse Instead of Rebuild

The solution leveraged two existing, battle-tested pieces of infrastructure: git diff and an internal CDN. The goal was simple: don't build what you already have.

Conditional builds. The team used git diff with 3-dot notation to compare the current branch against main, identifying the latest common commit. If the diff revealed any changes to the frontend, the pipeline invoked the standard build job. If not, the build step was skipped entirely.

Prebuilt asset lookup. When a fresh build wasn't needed, the pipeline located a recent production frontend build stored in AWS S3. An internal CDN served those prebuilt assets to the E2E testing environment, ensuring tests always executed against current, production-grade code without the overhead of producing a new artifact.

Scaling to a Monorepo

The approach was conceptually straightforward, but applying it inside a repository with over 100,000 tracked files introduced specific constraints. Each step had to hold up under the scale of dozens of daily merges and a massive file tree:

  • File tracking. Even in a monorepo this large, git could identify whether any frontend files had changed in just a couple of seconds.
  • Artifact management. With hundreds of PRs flowing through daily, finding a prebuilt version that was still fresh enough required consistent naming conventions and a balance between recency, storage cost, and retrieval speed.
  • End-to-end response time. The entire optimization—deciding a build was unnecessary and locating a valid, recent artifact—completed in under 3 seconds on average.

End to end pipeline depicting local pre-built frontends at the third stage.

The Results: Faster Builds, Fewer Builds, Less Storage

Eliminating redundant work produced across-the-board improvements:

  • 60% fewer frontend builds. Reusing prebuilt assets cut the weekly build count by more than half.
  • 50% faster builds. The average E2E frontend build dropped from roughly 5 minutes to about 2 minutes. Combined with a prior Webpack upgrade that brought builds down from ~10 minutes, total E2E pipeline time fell from ~10 minutes to ~2 minutes.
  • Multi-terabyte storage savings. AWS S3 usage dropped by several terabytes per month, eliminating duplicate assets that would have been retained for a full year.
  • Hundreds of hours saved. Both cloud compute time and developer wait time were dramatically reduced on a monthly basis.

January baseline was an average build time of 10 minutes and 6 minutes for testing. By August average build time was 2 minutes, an 80% decrease for the year.

Two Unintended Wins

The project surfaced valuable side effects beyond the headline metrics:

  • Lower test flakiness. Intermittent test failures, unrelated to code changes, dropped to their lowest recorded monthly level. The team attributes this to consistent asset delivery and avoiding the variability of complex on-the-fly builds.
  • A deeper codebase understanding. Tracing the optimization through legacy systems that hadn't been touched in years generated new insights and a backlog of potential enhancements for future work.

The Takeaway

The optimization didn't require novel tools or a rewrite of the CI/CD stack. It relied on re-examining a workflow that worked reliably but contained a hidden inefficiency. At its core, this project was about questioning a default behavior—building the frontend for every PR—even when the system never appeared to fail.

The implementation touched infrastructure written in Python, JavaScript, Bash, PHP/Hack, Rust, YAML, and Ruby, involving the build systems, pipeline orchestration, S3 storage, and the internal CDN, all without meaningful downtime to the deployment pipeline—save for a ten-minute hiccup that was quickly resolved.

For teams facing similar bottlenecks, the lesson isn't about frontend builds specifically; it's to scrutinize whatever step in a pipeline is treated as a fixed cost. Often, the most impactful performance gains come not from making each operation faster, but from determining which operations don't need to happen at all.