Replacing HAProxy With Envoy for Slack’s Websocket Tier

Slack’s real-time messaging depends on millions of simultaneously open websocket connections; at peak times, users push a high volume of small messages through these bidirectional links. For most of Slack’s history, HAProxy handled the load-balancing role for all inbound traffic, including the websocket endpoints that power typing indicators and instant message delivery. The websocket service (“wss”) is reachable via wss-primary.slack.com and wss-backup.slack.com (don’t bother visiting—you’ll only get an HTTP 404).

Websocket sessions begin as standard HTTPS requests and then upgrade via a protocol switch. Slack runs separate websocket services for messages, presence (who’s online), and other features; one dedicated endpoint exists for apps that need real-time Slack interaction. Previously, a fleet of HAProxy instances in multiple AWS regions terminated these connections close to users and forwarded requests to backend services.

HAProxy had served Slack since its early days, but operational constraints pushed the team to evaluate Envoy Proxy as a replacement. By 2019, Slack had already adopted Envoy as the data plane in its service mesh and had in-house Envoy expertise, making the move a natural standardization play: one load balancer across the entire ingress tier instead of maintaining two configurations, two build pipelines, and two sets of operational knowledge.

The Case for Moving Off HAProxy

Hot restarts and configuration churn. Backend endpoint lists at Slack change often as instances are added or cycled. HAProxy offers two ways to update its config: the Runtime API (which Slack used on another HAProxy set, with mixed results) or rendering backends into the config file and reloading. The websocket tier used the file-reload approach. Each reload spawned a fresh process for new connections while the old process lingered for hours to let long-lived websockets drain, avoiding user disconnections. But too many concurrent HAProxy processes meant the fleet couldn’t converge quickly on new configurations, forcing periodic reaping of old processes and rate-limiting reloads during backend churn. Managing this required extra infrastructure.

Envoy sidesteps these problems. It supports dynamically configured clusters and endpoints via EDS, so endpoint list changes don’t require any reload. When code or configuration do change, Envoy hot-restarts without dropping connections. It watches filesystem configs with inotify and copies statistics from the parent process to the child during restart, so gauges and counters survive the transition. The operational overhead drops significantly—no auxiliary services needed to manage reloads or config updates.

Richer load-balancing behavior. Envoy includes several advanced features out of the box:

  • Zone-aware routing
  • Passive health checking through Outlier Detection
  • Panic Routing—when the healthy-host percentage falls below a threshold, Envoy can route traffic to all backends, healthy or not. This proved valuable during Slack’s January 4, 2021 outage, which stemmed from a widespread network problem.

Beyond those, the broader goals were improved operability and standardization across Slack’s infrastructure.

Translating HAProxy Config Into Envoy

Envoy’s configuration model differs fundamentally from HAProxy’s. It revolves around four concepts: listeners (TCP/SSL/unix sockets that receive requests), clusters (internal services like message and presence servers), routes (which bind listeners to clusters), and filters (which act on requests).

Slack manages infrastructure configuration with Chef, and the initial approach—deploying Envoy config as a Chef template file—quickly became unwieldy. The team built Chef libraries and custom resources to generate Envoy configuration instead. Inside Chef, the configuration is a Singleton, reflecting that each host has exactly one Envoy config. Chef resources add listeners, routes, and clusters to that singleton, and at the end of a Chef run the complete envoy.yaml is generated, validated, and installed. Intermediate states are never written, preventing invalid configs from landing on disk.

Replicating the full HAProxy setup in Envoy took real effort, though most required features already existed—it was mostly a matter of adding Chef library support. Some missing Envoy features were contributed upstream; a few remain maintained in-house as extensions.

Testing and Validation Strategy

Validation was iterative. The team typically prototyped with hand-written Envoy configs on a development machine using one listener, route, and cluster each. Once a hand-coded config worked, it was folded into the Chef libraries.

HTTP routing was exercised with curl, covering:

  • Header- and cookie-based routing to specific backends
  • Path-, prefix-, and query-param-based routing
  • SSL certificate behavior

When behavior didn’t match expectations, Envoy’s debug logging (enabled via the admin endpoint: curl -X POST http://localhost:<envoy_admin_port>/logging?level=debug) revealed exactly why a request was routed to a given cluster. Debug logs are verbose and expensive—not suitable for production—but invaluable locally. The admin interface also exposed useful state:

  • /clusters: all clusters, upstream hosts, and per-host statistics
  • /certs: loaded TLS certificates as JSON, including serial numbers and expiration
  • /listeners: all configured listeners with names and addresses

Chef runs validate configs with --mode validate to prevent bad installs (sudo /path/to/envoy/binary -c </path/to/envoy.yaml> --mode validate for manual checks). Envoy’s JSON-formatted listener logs, ingested into Slack’s logging pipeline after PII sanitization, also aided debugging.

Gradual Production Rollout

To keep risk low, Slack stood up a parallel Envoy websocket stack with an equivalent configuration to the existing HAProxy tier. This permitted a gradual, controlled traffic shift and a quick path back to HAProxy if needed. The trade-off: double AWS resource costs during the migration, which Slack accepted for a transparent cutover.

DNS for wss-primary.slack.com and wss-backup.slack.com is managed via NS1. Weighted routing shifted traffic from haproxy-wss to envoy-wss NLB DNS names. Early regions rolled out slowly in 10%, 25%, 50%, 75%, and 100% steps over a week; later regions moved faster (25%, 50%, 75%, 100% within two days) as confidence grew.

The migration itself was outage-free but not without hiccups: minor differences surfaced around timeout values and headers. Several cycles of revert, fix, and re-rollout occurred along the way. After roughly six months, the HAProxy websocket fleet was fully replaced by Envoy Proxy across all regions—with zero customer impact.

Lessons from the Migration

By design, the migration was deliberately "boring" — uneventful and stable. That boring outcome was the goal, because excitement in load balancer migrations usually means something broke. Still, the process surfaced several lessons.

Envoy’s configuration model offers far more defined scopes than HAProxy’s. In HAProxy, one large configuration includes all listeners, which had allowed Slack’s config to grow organically into a tangle of accumulated rules. Envoy’s model scopes rules tightly to listeners and routes, making it easier to associate behavior with the correct requests. However, extracting what mattered from what was technical debt in the old HAProxy config took considerable time. It was often unclear whether a rule was intentional, accidental, or load-bearing for other services. For example, some services were meant to exist under only one of two virtual hosts, yet were available under both in HAProxy — and because existing code relied on that behavior, the migration had to faithfully replicate the mistake.

Several subtle behaviors in the HAProxy stack were missed, and some had real impact — Slack briefly broke its Daily Active User (DAU) metric. Minor issues were plentiful, but load balancer behavior is inherently complex, and time plus debugging was the only path through it.

Testing Gaps and Library Trade-offs

The migration started without an automated testing framework for load balancer configurations. There were no tests validating that test URLs routed to the correct endpoints or that request and response header behaviors were preserved — there was just the HAProxy config itself. Tests would have provided useful context about why certain behaviors existed, so the team frequently had to consult service owners directly to determine what they depended on.

The Chef resources built for Envoy intentionally supported only a subset of functionality, keeping the libraries simple and focused on actual usage. The trade-off was that any new Envoy feature required adding support to the Chef libraries first. SNI (HTTPS) listeners were added part-way through development because it was simpler than modifying existing listeners. Virtual host support was a different story — too much code was already in use across the company to refactor those resources. The vhost support in the Chef library remains a known hack, with a fix planned for the future.

To make changes to the Envoy Chef libraries safer, the team introduced a comprehensive test suite that generated the entire configurations of other teams using those libraries. This made the blast radius of any update instantly visible, showing exactly how all generated Envoy configurations would be affected.

Communication was another critical component. The Customer Experience team monitored incoming tickets for signs that users had been impacted, keeping everyone aligned throughout the rollout.

Standardizing on Envoy

The websocket migration was a success despite occasional setbacks. Slack has since migrated its software client metrics ingestion pipeline — isolated from other ingress load balancers — to Envoy Proxy. Migration of the internal load balancers for web and API traffic is nearly complete, and work is underway to move the regular, non-websocket HTTP stack that terminates incoming edge traffic from HAProxy to Envoy.

The end goal is now close: standardization on Envoy Proxy across both ingress load balancers and the service mesh data plane. This will reduce cognitive load and operational complexity while making Envoy’s advanced features available throughout the infrastructure. Since the migration, Slack has exceeded its previous peak load without issues.