Load Testing with Koi Pond

Why Slack built its own load-testing client

Complex systems resist prediction at scale. Slack’s architecture, where a single message can trigger asynchronous jobs and fan out events to millions of connected clients, is no exception. To find limits and surface bugs in a controlled environment, Slack engineers needed a load-testing tool that could realistically mirror client behavior without the cost of running full browsers.

A Slack client performs three core activities:

1. Sends API requests to the webapp backend, which may also enqueue asynchronous jobs.
2. Maintains a websocket connection for real-time events such as presence changes and typing indicators.
3. Sends API requests to Flannel, the application-level edge cache.

Consider the path of a simple chat.postMessage call. The connected client hits the Web API, the backend queries the real-time services stack for a timestamp, writes the message to the messages table in Vitess, kicks off an asynchronous job for link and attachment previews, and finally pushes an event through the real-time stack to every connected client in the channel. Replicating that chain of interactions with earlier tooling was the problem.

Previous attempts fell short

Slack had already tried two approaches. Puppet Show ran headless Chrome browsers pointed at slack.com across a Kubernetes cluster, exercising both backend systems and the real-time stack. It was accurate but expensive and slow to scale — reaching 150,000 simulated users required a full day of manual cluster setup.

API Blast hit a single API endpoint at a configured rate. It lacked the ability to test multiple endpoints simultaneously or exercise the real-time services via connected clients.

Those gaps defined the goals for Koi Pond: load test backend and real-time services cost-effectively, and realistically mimic scale-related phenomena such as thundering herds, message fan-out, and unbounded datasets.

Koi Pond: a slim client that swims in schools

Koi Pond runs lightweight versions of Slack clients, called koi. Each koi opens a websocket connection and makes API calls just like a real client. Koi are grouped into schools, each one a program managing up to 5,000 koi. Schools periodically poll a central server — the Koi Keeper — for assignments. Those assignments come in two forms: behaviors and formations.

Behaviors: realistic background traffic

Behaviors define actions koi take at a specified frequency, usually a single Slack API call such as conversations.history, though sequences can chain dependent calls. The configuration separates frequency from sequence, allowing the tool to scale a behavior to whatever number of koi are booted.

Frequencies derive from real production usage. Slack’s internal workspace data showed a 23.93% chance that a user would call conversations.history in a given minute; koi reproduce that rate. While stress testing to find breaking points has value, Slack found realistic pacing more useful for preparing systems for customer launches.

Behavior definitions are written using Golang’s standard template library and have a consistent structure:

• doc documents the behavior and its steps.
• pre_actions execute before the API call, such as selecting a random channel from the koi’s local store and halting if none is valid.
• request contains the actual network call with arguments drawn from real production usage.
• post_actions run immediately after, for example storing message and user IDs locally.

Writing these files by hand was tedious, so Slack automated the process with a dashboard backed by the data warehouse. Engineers enter a team ID and get a list of API frequencies ready to export and import into the tool, producing behaviors that closely reflect how actual workspaces operate.

Formations: coordinated swarms

Behaviors run on all koi at the configured rate; formations target a percentage of koi with a coordinated burst of outbound requests — API calls, websocket messages, or both — within a specified window. Formations simulate thundering herd scenarios, like an organization-wide channel flipping to an announcements feed with everyone reading and reacting simultaneously.

API Blast alone couldn’t capture that scenario. Calling reactions.add at a fixed rate tested the database and job queue but missed the downstream effect on the real-time stack, where reaction_added events would flood websockets across all connected clients.

A formation can chain realistic user steps: view a message, mark it read, then call reactions.add. Metadata specifies that 10% of booted koi complete the sequence within six minutes (a begin_within_secs value of 360). The tool’s name comes from the image of koi swarming in a pond, much like clients converging on a shared action simultaneously.

Monthly practice before a major launch

In the month preceding a large customer go-live, Slack combined Koi Pond with other validation methods to build confidence.

Pairing koi with manual QA

Manual QA engineers tested critical Slack features while koi provided background activity. Some client actions are too nuanced to replicate precisely yet, so human verification filled the gaps. Each week the test scaled up — starting at 5,000 koi and reaching 500,000 before the customer’s launch — while the workspace was shaped to match expectations for the real customer, including the right channel mix and EKM enablement. Those weekly sessions surfaced issues that graphs and alerts missed, with consistent iterations to verify fixes.

One such session uncovered a significant problem. With 400,000 koi active, a QA engineer manually shared a file. That action emitted two websocket messages per user, overwhelming the real-time infrastructure and forcing a shutdown to avoid production impact. Slack increased the splay of real-time servers for the test workspace to spread websocket events more evenly and optimized the file-upload code. Rerunning the scenario at 1 million koi produced barely a blip — a bug caught before it could trigger an incident with a real customer.

Koi Pond plus other load tests

Teams running API Blast or ad-hoc scripts could ask to have koi booted alongside them, revealing websocket and real-time server load that API-only tests missed. When one engineering team fired chat.postMessage at 50 requests per second with 200,000 koi connected, the real-time servers were overloaded — a clear signal for where to place a rate limit.

Koi Pond alone

Running only background behaviors without extra tooling still exercises the system realistically. Scaling to 200,000 koi strained the database namespace used by monitoring tools, prompting the monitoring team to scale up and manually add shards before the customer onboarding. The same setup proved useful for rolling out performance optimizations: once the load-test workspace handled a Vitess migration flawlessly, Slack felt confident enabling it for the new customer.

Formation-drawn guardrails

Coordinated formations against 1 million koi simulated a company-wide announcement. When 100 users reacted to the message and 18 thread replies arrived at roughly one per six seconds, the real-time infrastructure buckled. The takeaway: large-channel events need guardrails while underlying issues are addressed — a lesson learned before a real announcement caused an incident.

What Load Testing With Koi Pond Taught Us

After months of development and testing, Koi Pond has proven itself as a dependable tool for running massive, realistic load tests against Slack’s infrastructure. The results have been impressive on multiple fronts, and the lessons learned are shaping how we approach performance engineering going forward.

Key Successes

Koi Pond’s scalability is its standout feature. Our largest test to date simulated 2 million concurrent users in a single workspace, and we encountered no major issues or service incidents during that run. The system ramps up quickly and behaves in a way that closely mirrors real production traffic.

The cost efficiency is equally striking. Running 2 million users through Koi Pond costs roughly 0.26% of what it would cost to run 150,000 users with Puppet Show — that’s 13 times the number of clients for a tiny fraction of the price. This cost profile makes it feasible to run large-scale tests frequently rather than reserving them for rare, expensive exercises.

Safety was a primary design goal, and the architecture reflects that. Koi Pond is wired directly into our emergency-stop mechanism, and the entire system can be shut down within seconds through the Pause and Stop controls, which are surfaced prominently in the UI. The load-test infrastructure is also fully isolated from production: it runs on its own web tier and uses a dedicated VTgate proxy to route test traffic, keeping any potential issues contained.

The payoff came during a major customer launch. Because we had already validated the system under extreme load, deployment day was smooth and uneventful — which is exactly what we wanted. The absence of surprises was a direct result of thorough preparation.

Hard-Won Lessons

The path wasn’t without its setbacks, and the challenges revealed important principles for anyone building similar tools.

Scale gradually. During one preparatory phase for a customer launch, we jumped straight from 5,000 to 100,000 koi. That sudden spike put strain on one of our databases, and the incident could have been avoided by staging through intermediate load levels and observing the incremental impact on system health.

Realism is difficult to achieve. Slack’s architecture is complex and highly distributed, with nuances that are hard to replicate in a test environment. For example, in an SSO (single sign-on) test combining API Blast with Koi Pond, we initially used a single SSO token at a high request rate. No websocket events came through, which was a red flag. Switching to multiple users’ SSO tokens produced the expected volume of websocket traffic and allowed us to properly exercise our real-time servers. Without that realistic setup, we would have missed an entire class of load on those systems.

Collaboration with product teams is essential. Several teams volunteered to test Koi Pond on their own features, and their feedback helped us refine the tool to be more approachable and practical. Their input identified friction points we hadn’t anticipated and improved the overall workflow for everyone.

High fidelity takes time. Achieving a truly realistic simulation remains a work in progress. Distributing users across channels and implementing every API we have is time-consuming. Simulating all the product’s flows and nuances accurately is one of the hardest parts of this work, and we’re still iterating on it.

Where Koi Pond Goes Next

The tool has already enabled proactive, realistic testing across our systems without breaking the bank, and it played a decisive role in the successful launch of a large customer. But we see two clear areas for improvement.

First, we want to run koi continuously. Right now, every test requires booting up all clients from scratch, which adds significant overhead when other teams want to use the tool. A persistent pool of running koi would eliminate that startup cost and open the door to new use cases, such as load testing on every deploy and running tests that better reflect timezone and regional traffic patterns.

Second, we aim to make behaviors more flexible and higher fidelity. Each test currently runs a single configuration modeled on one production workspace. Real-world Slack usage varies by timezone, region, and client type — mobile versus desktop, for example. Building support for these patterns into the tooling would make each test that much closer to the real thing.