Load testing around the clock
Most load-testing setups follow the same rhythm: spin up test clients, run a burst of traffic, tear everything down, then analyze the results. That model works, but it creates real friction. Engineers have to reserve time and infrastructure, and the tests only capture a single moment in time. Slack’s infrastructure team took a different path — instead of load testing occasionally, they decided to load test continuously.
The idea came from a slate of goals: build a store of data that reflects how Slack’s largest customers actually use the product, make it easy for product and infrastructure teams to run load tests regularly, and fold load testing into the release cycle. Continuous load testing checked every box. With test clients always running, engineers don’t need to spend time booting clients or configuring environments. When a new build hits production, tests can run immediately against a large, active organization to catch regressions. And because issues surface incrementally, there’s no need to rally teams ahead of a single test event.
The urgency increased with an upcoming launch: GovSlack, a version of Slack run in GovCloud for public sector customers with stringent security requirements. Before making continuous load testing permanent, the team needed a platform that was safe, resilient, and understood by other engineers.
How the pieces fit
Koi Pond is Slack’s primary load testing tool. A “koi” is a slimmed-down Slack client that makes two types of network requests: API calls and messages over a websocket connection. The requests a koi makes are governed by a configuration file built from average user behavior, courtesy of Slack’s data warehouse.
Koi run in Kubernetes pods called Schools, maxing out at 5,000 koi per School. The Keeper — the Koi Pond server — tracks test details like which organization is being tested and how long the test lasts. Schools poll the Keeper regularly for work assignments or status updates.
Safety first
The load test environment is mostly isolated, but some infrastructure is shared with production traffic — which meant safety features had to come first.
One addition was a service called Automatic Shutdown. It polls metrics and, when a value falls outside a defined threshold, sends a signal to stop active load tests. The service uses Trickster APIs to query Prometheus, and operators define thresholds via a list of Query objects that include the query string, time range, Prometheus cluster, value threshold, comparator, and label.
As an example: if web API success rate drops below 95% for five minutes, all active load tests are shut down. Automatic Shutdown runs from Slack’s shared Go repository, with input formats designed so adding new queries is trivial. Because tests register with the service, a load test can’t add strain to an unhealthy system or obscure a genuine production incident. An added benefit: fewer off-hours pages, because manual intervention is no longer required to stop bad traffic.
Koi Pond’s UI also gained the ability to partially shut down a load test. If traffic looks suspect, operators can remove a specific percentage to isolate the source of the problem. And Emergency Stop — a service every load test at Slack registers with — now has a UI showing all running tests and a history of past emergency stops. Any engineer can trigger it, particularly during high-severity incidents.
Retaining state and automating tokens
Koi Pond began as an ultra-lightweight service with all load test data stored in memory. That was fine for one-off tests, but continuous operation demanded something more durable.
Kubernetes doesn’t guarantee pod uptime. Mandatory updates, security patches, and new deployments all kill and replace pods, and when the Keeper pod restarted, all its memory of ongoing tests vanished. The solution was to add a database — specifically, AWS DynamoDB. Its NoSQL model gives Koi Pond a flexible, dynamic schema and supports the unstructured data (token configurations, formation scripts, behavior files) the system stores as documents or JSON.
With state persisted, the team also tackled token generation. Koi Pond uses user tokens to mimic real human action, and those tokens need periodic regeneration when they expire. The original process was manual: an engineer edited a configuration file, ran a multi-hour script, exported output to the container base image, and deployed. That workflow fit when tests covered 15 workspaces and 500K users — but load testing now spans different organization shapes, including some with thousands of workspaces, and the team has grown. Enterprise and authentication product teams use the tool more, which means session-expiration scenarios requiring more frequent token generation.
The new process automates token generation with a cron job. It batches users to the job queue for asynchronous token generation and storage — tokens go to an encrypted S3 bucket rather than the container base image. Individual job failures retry automatically with exponential backoff, while fatal errors surface to an engineer, who can retry via a UI after debugging. Built-in logging gives visibility into errors. Automating a key requirement for load testing made the whole pipeline more reliable.
A deliberate rollout
For the release itself, communication was critical. Noisy alerts or an unstable tool would erode trust in load testing across the organization. The plan called for gradually ramping from 5,000 koi to 500,000, with a week of monitoring at each step.
But groundwork started long before. The team worked with infrastructure groups — asynchronous services, data stores, real time services, and Flannel — from the beginning, asking core teams to review the proposal before any technical work began. That review window gave partners a chance to provide feedback that went into the tech spec.
Cost was another open question: What does it cost to run a large number of clients around the clock? Projections came from the Cloud Economics team based on one-off tests, and the team committed to monitoring closely to avoid an out-of-control AWS bill. Koi Pond proved light enough to keep costs well within expectations.
On-call burden mattered too. The team is just four engineers, so they wanted incident and deploy commanders well equipped as a first line of defense. Communication began early with incident command rotations, and various metrics were added to the deploy health dashboard, including load test API success and latency, gateway server health, and the percentage of 5xx errors on the load test tier. Comprehensive documentation covered how to handle potential issues.
Rollout notifications went to all relevant infrastructure teams and channels such as #deploys and #ops, sent via workflows.
Each message pointed to Emergency Stop processes and relevant dashboards. Combined with the safety features built into Koi Pond itself, the launch was smooth — some might even say anticlimactic. In the six months since continuous load testing went live, the team hasn’t been paged off-hours once.
What Continuous Load Testing Has Bought Us
Running our load-testing clients non-stop against an org four times the size of our largest customer has created a persistent dataset that serves as a performance baseline for very large workspaces. That baseline has made it possible to validate broad-impact changes—like HHVM upgrades and auth modifications that touch every API—before sending them to production, without worrying about customer-facing performance degradation.
The infrastructure has also made load testing a routine part of feature development for product and engineering teams. For instance, our Cloud Infrastructure team used the continuous clients to simulate removal of traffic from a single availability zone and observe system and monitoring behavior without any risk to real traffic. The Messaging team, meanwhile, used it to validate the efficiency of cached user profile lookups against a set of new tables they’d created.
Event preparation has gotten faster as well. For the GovSlack launch, engineers got realistic performance signals in minutes once we added the ability to point Koi Pond at the GovSlack environment. Previously, standing up an ad-hoc test of comparable size would have consumed a full day of engineering effort.
An Incident That Escaped the Net
Continuous load testing can’t catch everything. A large customer’s admin once posted to an org-wide default channel and users hit degraded performance. We were baffled initially because we’d tested a similar case at 4x the traffic. The difference: we hadn’t accounted for the volume of reactions on earlier posts that every user had to fetch when the channel loaded.
We deployed several fixes, and here’s where the persistent setup paid off. Reproducing that exact scenario would have taken about a week of ad-hoc prep before, or we might have waited for the customer to trigger it again. With Koi Pond, we added a few lines to the configuration for already-running clients, populated the channel as needed, and recreated the incident in roughly an hour. Verification of the fixes followed immediately, and the customer’s next org-wide message went out without issues.
Adoption and Release Integration
We’ve held Koi Pond learning sessions and produced video tutorials now included in engineering onboarding. That outreach helped boost our feature coverage by 10%. Load testing has become part of the release workflow, and we’re working on making it more actionable. For example, automated QA tests focused on performance run five times a day against the load test org. Whereas Koi Pond metrics catch regressions in backend code, this framework identifies client-side bugs. We plan to make those tests blocking in the deploy pipeline so performance bugs can’t reach production.
Koi Pond’s alerting has already spotted several regressions ahead of release—including experiments that degraded performance only for large orgs and problematic hot cache keys—and has sometimes surfaced early signals for incidents before they fully erupted.
Next Steps
We’re continuing to refine the system. On the immediate roadmap: automatically updating behaviors from our data warehouse, expanding the set of Koi Pond tests in the deploy pipeline, and smoothing the experience for engineers authoring load tests.
