Escaping the Capacity Trap: How Slack AI Rebuilt Its Inference Layer

When Slack began serving Large Language Models to enterprise customers in early 2023, the engineering challenge was about more than model quality. The system had to meet strict security, reliability, and performance requirements while remaining resilient to regional outages and GPU supply shortages. What followed was a three-year architectural evolution through four distinct phases, each driven by a specific operational bottleneck.

Phase One: The Managed Service Tax

The first iteration of Slack AI ran on AWS SageMaker. It satisfied the initial security requirements out of the box: FedRAMP compliance, a zero-knowledge escrow VPC, and a clean separation between Slack's data and the model provider's proprietary weights. Containers were deployed across multiple AWS regions to maximize uptime, which meant building custom tooling for cross-region IAM roles, model endpoint routing, and auto-scaling.

That custom tooling quickly became the problem. Three operational costs dominated:

  • Slow scaling: Container initialization times made instantaneous scale-out impossible.
  • Hardware scarcity: Enterprise-grade GPUs, including A100 and H100 instances, were frequently unavailable.
  • Over-provisioning: Idle capacity had to be maintained to meet peak performance SLAs.

On-Demand Capacity Reservations and cron-based scaling helped by early 2024, but the team was still spending most of its cycles on infrastructure plumbing rather than model quality. A second issue compounded the problem: AWS was prioritizing feature development on Amazon Bedrock, its managed LLM service. New model optimizations repeatedly appeared first on Bedrock, causing a lag on SageMaker that mattered for a product competing on model quality.

Phase Two: Bedrock and the Provisioned Throughput Trade-Off

By mid-2024, Bedrock had reached FedRAMP Moderate, and migration became the obvious next step. The move eliminated the hardware abstraction problem immediately. Slack stopped managing GPU instance lifecycles and instead purchased throughput in the form of Model Units, each providing a deterministic rate of tokens per minute.

The migration was executed with deliberate caution. Legal, Security, and FedRAMP approvals were secured before production traffic moved. Extensive load tests mapped the exact number of Model Units needed to match the previous baseline across traffic profiles. A/B testing verified output quality, and gradual traffic shifts via feature flags provided instant rollback capability. The transition finished with zero customer-facing incidents.

The operational and product benefits landed quickly. Engineering teams moved from reactive capacity management to strategic forecasting, projecting demand weeks ahead. AI Search was upgraded to newer high-reasoning models much sooner than the old infrastructure would have allowed.

Nevertheless, two efficiency gaps remained after the migration:

  • Peak-driven over-provisioning: Global traffic peaks during US morning hours forced a high baseline of Model Units, even when APAC and EU usage was light or it was a weekend. Those troughs represented persistent unused capacity.
  • Commitment lock-in: Provisioned Throughput required commitments of one to six months, which became a liability in a model landscape where state-of-the-art capabilities can be superseded within weeks. Slack often delayed model upgrades until existing commitments expired.

Those two problems, a fixed capacity bill for variable demand and a contractual drag on innovation, drove the search for an alternative balancing reliability with On-Demand flexibility.

On-Demand Capacity Ends the Idle-Resource Problem

After validating Bedrock and building mature monitoring, we targeted the last efficiency gap. Usage analysis showed feature demand tracks business hours, leaving capacity idle overnight. A static footprint sized for 24/7 peaks was wasteful.

Moving to on-demand infrastructure eliminated the idle capacity problem and delivered the architectural agility needed for highly variable workloads. For features with a 10x variance between peak and off-peak usage, the efficiency gains were significant. More importantly, this freed us from multi-month commitments: as soon as a better model passed our quality bar, we could migrate features within a day instead of waiting for contract expiry.

Hybrid Routing Balances Performance and Cost

We didn't flip everything to On-Demand at once. A Hybrid Routing strategy kept high-volume, latency-sensitive features on dedicated Provisioned Throughput for a consistent feel, while asynchronous and bursty jobs like nightly Recaps moved to on-demand. A Spillover Pattern bridged the two: excess requests automatically routed to on-demand endpoints, so we never dropped traffic due to capacity ceilings.

On-Demand traded rigid planning for flexibility and removed manual capacity management. Bedrock's ability to route across US regions based on real-time availability let us find capacity dynamically while respecting regional data boundaries. The trade-offs had to be engineered around:

  • Service Level Variability: On-Demand runs on a shared-resource model with different uptime characteristics than dedicated capacity.
  • Regional Capacity Orchestration: Success depends on the provider managing demand across its entire regional customer base, not hardware reserved for Slack.
  • Concentration Risk: Heavy reliance on one provider's on-demand pool means a service-wide incident could impact all Slack AI features at once.

Intelligent Fallbacks Strengthen Resilience

To mitigate these risks, we built a model hierarchy for every AI feature. When the primary model degraded—due to elevated time-to-first-token, throttling errors, or declining customer feedback—the platform automatically rerouted to a fallback model. A complex failover remained invisible to customers, who continued receiving high-quality results seamlessly.

This fallback logic revealed two strategic gaps. First, engineering failovers within one cloud still left us exposed to a provider-wide outage. Second, the AI landscape is fragmented and moves quickly: the leading model for summarization, reasoning, or high-speed extraction often changes within weeks and is frequently exclusive to a specific cloud. Any single-vendor dependency risked limiting access to the best available technology while we work to meet security, compliance, and privacy standards.

Expanding to a Multi-Cloud Ecosystem

Enterprise-grade reliability and a best-of-breed model strategy required going beyond any single provider. By early-2026, we added Google Cloud Platform (GCP) Vertex AI—not just as a failover, but as a strategic engine broadening our catalog of state-of-the-art models. Four factors drove the shift:

  • Infrastructural Redundancy: For a mission-critical Digital HQ, uptime is the primary metric. A multi-cloud footprint removes provider-level disruption as a single point of failure; traffic reroutes to a healthy stack without interruption.
  • Model-to-Feature Optimization: Matching a model's strengths to specific feature requirements produced immediate gains: ~10% higher quality on complex reasoning and ~67% lower latency for high-velocity, low-token workloads.
  • Access to Innovation: Frequent vendor exclusivity means the latest breakthroughs may appear on any cloud. Multi-cloud keeps us ready to integrate new models while preserving our compliance, privacy, and security commitments.
  • Dynamic Workload Orchestration: Beyond failover, multiple providers enable sophisticated traffic shaping that evaluates provider health and performance profiles in real-time.

Building the Integration

Standing up production-ready GCP integration required tight coordination across Security, Risk and Compliance, Trust and Integrity, AI Quality, Legal, and Cloud Providers to keep data boundaries ironclad. We solved cold-start engineering challenges with secretless authentication and an API Normalization layer that converts disparate provider signals into unified application logic.

Intelligent Routing as the Core Abstraction

The central engineering challenge was abstracting away provider complexity. Our enhanced Intelligent Routing Layer guarantees users get the fastest, highest-quality response available; when a model or provider slows, the system reroutes instantly without any visible impact. The layer includes:

  • Metric-Driven Model Selection: Internal quality metrics determine the optimal model per feature—benchmarks showing one LLM superior for "Recaps" directs traffic accordingly. Every feature designates backup models; if the primary fails real-time quality or performance thresholds, routing automatically moves to the next choice.
  • Experimental Rules & A/B Testing: Testing a new LLM for Recaps—after security and compliance verification—meant routing a percentage of traffic to the new model with minimal code changes. Validation in the wild tightened our feedback loop from weeks to days.
  • Automated Circuit Breaker: Replacing manual failovers, this watchdog monitors endpoint health signals like Time to First Token (TTFT), 5xx errors, and latency p90 crossings. When the breaker trips, traffic diverts to a healthy alternative model by use case and complexity. The breaker enters a partial-open state, trickling controlled requests to the degraded endpoint, and expands traffic incrementally as health returns—ensuring graceful recovery without overwhelming a stabilizing service.

Managing Multi-Cloud Complexity

Operating a multi-cloud footprint at this scale is an acknowledged trade-off:

  • API and Behavioral Friction: Each provider's API patterns, error codes, and rate-limiting behaviors differ. Our normalization layer translates "Rate Limit Exceeded" from one and "Throttling Exception" from another into identical application signals.
  • Operational Monitoring Complexity: Native dashboards per cloud created blind spots. We built a unified monitoring stack integrating telemetry from both clouds so on-call engineers can diagnose issues without switching consoles.
  • The Attribution Challenge: Tracking per-feature cost gets harder as workloads shift dynamically between clouds. Deep instrumentation across billing systems maintains financial transparency.
  • The On-Call Knowledge Gap: Engineers must now be provider-agnostic, with expertise across infrastructure patterns and networking nuances of multiple clouds. Troubleshooting a distributed, multi-vendor footprint requires broader skills.

The operational overhead is justified by a superior service: single points of failure are gone, quality benchmarks improve by matching models to features, and we gain strategic leverage to adopt new innovations the moment they appear.

Lessons Learned on the Road to Multi-Cloud

Reaching a multi-cloud architecture wasn't about adding complexity—it was about strengthening Slack’s commitment to innovation and reliability. Looking back, five key principles guided that journey.

Cross-Functional Alignment Is the Foundation of Scaling

The real bottlenecks in scaling AI were rarely purely technical. Legal, risk, compliance, and security processes demanded as much attention as the code. Deep coordination between those teams and engineering is what made it possible to serve millions of users while preserving trust standards.

The Abstraction Layer Determines Agility

During the Phase 2 migration, the decisive factor wasn’t the choice of model—it was the logic wrapped around each model. That abstraction layer is what gives engineering the freedom to move quickly, which remains the primary competitive advantage.

Architecture Should Be a Living Document

Managed services improve rapidly, sometimes month to month. By staying provider-agnostic, Slack can integrate improvements in latency or reasoning without rebuilding from scratch. A flexible posture turns market progress into incremental upgrades.

True Reliability Means No Single Provider Dependency

Internal failover alone doesn’t cover every scenario. The Phase 4 move to a multi-provider stack was specifically designed to keep Slack operational even during a broad cloud outage affecting an entire platform.

Rethinking What Counts as Failure

An LLM service that is technically “up” but slow is effectively down. By classifying metrics like p90 spikes as soft failures and tracking feedback trends, the routing layer can respond proactively—ensuring users experience a responsive system, not just an available backend.

The direction of enterprise AI points toward multi-cloud, multi-model environments with dynamic orchestration. By prioritizing portability and staying close to market shifts, Slack has done more than integrate AI—it has built a foundation that adopts the industry’s best advances the moment they appear.