Scale, Stale State, and an HAProxy Slot Shortage

On May 12, 2020, Slack experienced a significant service outage that began at 4:45pm Pacific time. The root cause traced back to a combination of a morning database incident, aggressive autoscaling, and a longstanding bug in the tooling that syncs service discovery data with HAProxy's runtime state.

The Morning Incident

The trouble began around 8:30am with alerts from the Database Reliability Engineering team about a significant load increase. The Traffic team simultaneously flagged failures on some API requests. The cause was a configuration change rollout — specifically a percentage-based feature flag — that triggered a previously latent performance bug. The flag was quickly identified and rolled back, limiting customer impact to roughly three minutes.

That brief incident had a lasting side effect: it caused the webapp tier to scale up substantially. Because of increased usage during the pandemic, the fleet was already running far more instances than in early 2020. During the morning event, workers sat saturated while waiting on slow database requests, which drove utilization up and triggered autoscaling. The webapp tier grew by 75% during the incident, reaching the highest number of hosts Slack had run to date.

An 8-Hour-Old Configuration Bug

For the next eight hours, the system appeared healthy until alerts indicated an abnormal number of HTTP 503 errors. The webapp on-call engineer manually scaled the fleet up as an initial mitigation, but unlike the morning incident, this had no effect. Engineers quickly noticed that a subset of the webapp fleet was heavily loaded while the rest sat idle.

Investigation focused on both webapp performance and the load balancer tier. Slack routes webapp traffic through HAProxy instances, which are positioned behind a layer 4 load balancer. Service discovery relies on Consul and consul-template to render lists of healthy backends for HAProxy. However, Slack does not write host lists directly into the HAProxy configuration file, because doing so would require a reload for every change — and frequent reloads are costly.

Figure 1: High-level view of Slack’s ingress load-balancing architecture

Instead, HAProxy server state is manipulated at runtime via the Runtime API. Slack defines HAProxy server templates as "slots" that backends can occupy. When an instance is provisioned or becomes unhealthy, the Consul catalog is updated and consul-template renders a new host list. A custom tool, haproxy-server-state-management, reads that list and updates HAProxy state through the Runtime API.

Figure 2: How the set of webapp backends is managed on a single Slack HAProxy server

The Musical Chairs Problem

HAProxy is configured with N slots per AWS Availability Zone across M parallel pools, yielding N*M total backend slots. This had always been ample headroom — until the morning's database incident pushed the running webapp instance count slightly above the total slot capacity.

That was not immediately fatal, as there was still sufficient serving capacity. The problem emerged over the course of the day due to a bug in the server-state sync program. The sync tool always attempted to find a slot for new instances before freeing slots taken up by old instances that were no longer running. Once all slots were occupied, the program began to fail and exit early. Running HAProxy instances therefore stopped receiving state updates. As the webapp autoscaling group expanded and contracted through the day, the backend lists in HAProxy became increasingly stale.

By the afternoon, most HAProxy instances could only route traffic to the set of backends that had been alive since the morning — which had become a minority of the fleet. Newly provisioned HAProxy instances had correct configurations, but the bulk of the fleet was more than eight hours old, stuck with full and outdated backend state. The user-visible outage was triggered when the webapp tier began to scale down at the end of the US business day. Since autoscaling prefers to terminate older instances, the remaining old backends no longer matched the number still present in HAProxy state, leaving too few valid routing targets to meet demand.

Figure 3: ‘Slots’ in the HAProxy process, with some excess webapp instances that aren’t receiving traffic

Resolution and Root Cause

Once the cause was identified, mitigation was straightforward: a rolling restart of the HAProxy fleet. Slack's monitoring had alerting designed to catch this exact condition, but it had not been functioning properly. The alerting gap went unnoticed largely because the system had operated for a long time without modification. The HAProxy deployment itself was relatively static, so few engineers were actively touching its monitoring and configuration.

Figure 4: HAProxy state has grown stale over time and references mainly deprovisioned hosts

Slack attributes the relative neglect of this stack to an ongoing migration toward Envoy Proxy for all ingress load balancing. Websocket traffic has already moved to Envoy. While HAProxy has performed reliably for years, operational sharp edges of this kind make the transition attractive. Envoy's native integration with an xDS control plane for endpoint discovery eliminates the custom pipeline required to manipulate HAProxy state. Slack notes that recent HAProxy versions (since 2.0) resolve many of these operational issues, but Envoy is already the established proxy choice in their internal service mesh, making it the clear direction for ingress traffic. Early testing of Envoy with xDS at scale has been promising, and the architecture is not susceptible to this class of failure.