Deploying a monolith hundreds of times a day
Slack’s primary service, known internally as “The Webapp,” receives hundreds of changes from hundreds of developers each week. That scale makes large, one-off releases impractical. Two goals drive the approach: get new work in front of customers quickly for fast iteration and feedback, and avoid releasing a large batch of changes at once—big releases increase the chance of errors and make them harder to isolate during debugging.
A bot in the driver’s seat
Slack’s Webapp repository is deployed to production 30–40 times daily, with a median of 3 pull requests per deploy. Managing that cadence falls to ReleaseBot, an automated system now running 24/7. That was not always the case. Previously, deployments were supervised by a rotating crew of "Deploy Commanders" (DCs): developers pulled from the Webapp team who worked two-hour shifts guiding deployments, checking dashboards, and running manual tests.
The problem wasn't tooling or documentation—it was confidence. DCs who interact with a large system every few months struggle to know what “normal” looks like, what to do when something breaks, and when to stop a release. The Release Engineering team heard this feedback consistently.
The root issue was that humans weren't great at making high-level “go/no-go” calls during a deploy. Release Engineering initially developed the ReleaseBot not as a full automation project, but as a way to give DCs clearer signals. After running alongside human commanders for a quarter, it became apparent that the bot was faster and more consistent at catching problems than its human counterparts. It earned the driver's seat.
The payoff comes in two forms. First, with monitoring done right, computers are more vigilant than humans. Second, the automation frees engineers from staring at dashboards—a constrained, valuable resource better spent elsewhere.
Monitoring fatigue isn't a prerequisite
Any on-call engineer knows the cycle: start with tight thresholds over everything, get paged too often due to noise, delete alerts and loosen thresholds to get some sleep, then an incident slips through unmonitored. Incident reviews follow, and the next round of tightening begins. This treadmill stops many teams from even attempting automated deploys. The common retort is "what if something breaks?"—and the conversation never reaches an answer.
Continuous tuning of alerts happens because complex systems are naturally noisy. Background errors, changing performance characteristics, and shifting dependencies make absolute thresholds a judgment call: is 100 errors a problem today? What about 200 ms average latency? It’s hard to re-validate those numbers every month, let alone write code that knows the answer.
This framing makes automation feel insurmountable. Yet deployments are actually easier to guard than steady-state operations—provided you compare versions rather than set thresholds.
Nothing changed? Then it’s a good deploy
Error counts in a steady state aren't always meaningful signals for a deployment. If version 1 and version 2 both emit 100 errors per second, then the new version introduced no breakage. The goal is anomaly detection: watching for things that are tangibly different between the running old version and the new one.
Humans are already good at watching deployments this way. When you scan a dashboard after a deploy, you don't memorize thresholds for every graph—you instantly notice the spike. Even without knowing what a metric means, a spike tells you the state of the system shifted.
Teaching a computer to do this avoids the burden of defining “bad” in absolute terms. An anomaly is a statistically untangled signal: a new version showing an anomalous error rate is suspect, even if nothing else is known about the system. Determining "anomalous" is a statistical problem, while setting hard thresholds is a judgment call filled with tradeoffs between under- and over-alerting.
This insight has a practical implication: the metrics you need to automate deployments are likely the ones you already collect. Using hard thresholds during a deploy is still reasonable; those numbers are at hand and cover most cases. But anomaly detection provides signal quality on par with an experienced engineer scanning a dashboard.
Anomaly detection, two ways
ReleaseBot relies on two methods for spotting anomalous behavior in metrics: z scores and dynamic thresholds. The first catches statistically unusual spikes; the second filters out routinely expected noise.
Z scores: spotting the unusual jump
A z score tells you how many standard deviations a data point sits from the mean. The higher the absolute value, the more of an outlier it is. It's a way to mathematically identify a spike on a graph that might otherwise go unnoticed.
ReleaseBot is a Python application, and it uses scipy.stats.zscore to compute these scores for every data point in a given interval. The math is computationally cheap.
from scipy import stats
def calculate_zscores(self) -> list[float]:
# Grab our data points
values = ChartHelper.all_values_in_automation_metrics(
self.automation_metrics
)
# Calculate zscores
return list(stats.zscore(values))Most monitoring tools have built-in functions for the mean and standard deviation, so you can achieve the same result in Prometheus or Graphite. Here's the equivalent for the last five minutes of data in PromQL:
abs(
avg_over_time(metric[5m])
-
avg_over_time(metric[3h])
)
/ stddev_over_time(metric[3h])When a z score breaches its threshold, ReleaseBot automatically halts any in-progress deployments and alerts the appropriate Slack channel. For the majority of metrics, ReleaseBot uses a threshold of 3 or -3 (the latter catches drops). A z score of 3 typically corresponds to a data point above the 99th percentile, though the exact percentile depends on your data's distribution.
A z score of 3 doesn't require a large absolute change in the metric's value. If a graph has hovered between 1 and 3 for three hours, a jump to 5.5 yields a z score of 3.37—a clear breach. In absolute terms, the increase is only 2.5, but it's a massive statistical outlier. A static threshold would have missed it entirely.
>>> from scipy import stats
# List representing a metric that alternates between
# 1 and 3 for 3 hours (180 minutes)
>>> x = [1 if i % 2 == 0 else 3 for i in range(180)]
# Our most recent datapoint jumps to 5.5
>>> x.append(5.5)
# Calculate our zscores and grab the score for the 5.5 datapoint
>>> score = stats.zscore(x)[-1]
>>> score
3.377882555133357
A few practical notes from our experience:
- Start with a threshold of 3. It's a good default for most metrics.
- Handle a standard deviation of zero. If all your data points are identical, the equation divides by zero.
scipy.stats.zscorereturnsnanin that case; ReleaseBot overwrites it with 0, treating a flat line as normal. - Consider direction. You may only care about increases for some metrics and decreases for others. Think about whether a drop in errors is actually a signal.
- Monitor unconventional signals. ReleaseBot watches total log volume, for example. A surge in informational logs usually isn't worthy of paging an on-call, but it can signal unexpected behavior during a deployment.
- Make snoozing easy. When a change is anomalous historically but you know it's the new normal, you need a way to silence it. ReleaseBot calculates z scores over the last three hours, so its UI offers a "Snooze for 3 Hours" button for each metric.
How Slack interprets z score alarms
Slack treats z scores as high-confidence signals—they indicate something has definitely changed. The urgency is communicated with a standard color system in Slack messages: white for the lowest urgency, blue for medium, red for the highest.
A single z score breach is a blue circle. It's a warning that warrants investigation, not immediate panic. Multiple simultaneous z score breaches are a red circle—when several graphs jump at once, something is genuinely wrong, and remediation actions are justified before a full root cause analysis.
Beyond the typical golden signals (errors, 500s, latency), we've found some less obvious metrics useful:
| Metric | High z score | Low z score | Notes |
| PHPErrors | 1.5 | – | We choose to be especially sensitive to error logs. |
| StatusSlackCom | 3 | -3 | This is the number of requests to https://status.slack.com – the site users access to check if Slack is having problems. A lot of people suddenly curious about the status of Slack is a good indication that something is broken. |
| WebsocketEventsVolume | – | -3 | A high number of client connections doesn’t necessarily mean that we’re overloaded. But an unexpected drop in client connections could mean we’ve released something especially bad on the backend. |
| LogVolume | 3 | – | Separate from error logs. Are we creating many more logs than usual? Why? Can our logging system handle the volume? |
| EnvoyPanicRouting | 3 | – | Envoy routes traffic to the Webapp hosts. It starts “panic routing” when it can’t locate enough hosts. Are hosts stopping but not restarting during the deployment? Are we deploying too quickly – taking down too many hosts at once? |
Dynamic thresholds for known patterns
Static thresholds still have a place in ReleaseBot, but they're considered low-confidence alarms (white circles). For key metrics, ReleaseBot calculates its own dynamic threshold and uses whichever is higher.
Consider a scenario where a database team deploys every Wednesday at 3pm, causing a temporary spike in database errors that your application handles gracefully. Users don't notice, and you don't want to stop deployments over it. A dynamic threshold can accommodate this by using an average from historical data.
The key is to sample from *similar* time periods, not just a continuous window. Since Slack is used primarily during the typical workday, at 6pm on a Wednesday, ReleaseBot pulls data from:
- 12pm–6pm Wednesday (today).
- 12pm–6pm Tuesday.
- 12pm–6pm last Wednesday.
These windows are pooled to calculate a simple average. In PromQL, it looks like this:
(
sum(metric[6h])
+ sum(metric[6h] offset 1d)
+ sum(metric[6h] offset 1w)
) / 3The algorithm is straightforward:
- Calculate the average of the pooled historical data.
- Take the greater of that average and the hard-coded threshold.
- Alarm and stop deployments if the last five data points breach the chosen threshold.
In practice, this means ReleaseBot watches the thresholds but is willing to ignore a breach when historical data indicates the behavior is routine. Dynamic thresholds are a refinement rather than a requirement—static thresholds may be noisier, but they carry no additional risk to production.
Facing the fear of automation
Being afraid to break production is a legitimate reason many teams hold back from automating deployments. The key is recognizing that monitoring for automated deploys differs from day-to-day system monitoring, and that simple tools can address it.
Slack took a careful, iterative path. After building z score monitoring into ReleaseBot, the team compared its results against humans manually running deployments and watching dashboards. ReleaseBot's performance was so strong that it felt irresponsible to keep humans in the driver's seat.
Put some z scores on a dashboard and see what they catch. You might find your team spends a lot less time staring at graphs.
