Bounding Goroutines for Predictable Throughput
Running a pipeline that handles more than a billion events a day exposes scaling faults quickly. Shopify's Server Pixels service processes storefront events such as checkout completion, search submission, and product view — but only for customers who have opted into tracking. An increase in traffic exposed a design flaw: the service was spawning goroutines without limit. The fix was a bounded worker pool, which improved per-pod processing throughput from 7.75 thousand events per second to 21 thousand events per second.
Server Pixels is a Go service that validates, processes, and augments events before forwarding them to marketing partners via Kafka. When event volume increased, so did consumption lag on the input topic. And when any downstream component slowed, the service risked exceeding its five-minute processing SLO. The underlying cause was straightforward: every batch of events created new goroutines with no cap on concurrency. Under heavy load, the machine ran out of CPU and memory just managing goroutine scheduling.
Anticipating three times the usual traffic for Black Friday Cyber Monday (BFCM), the team needed a design that would use resources more efficiently rather than relying on vertical scaling.
Channel as a Task Queue
The worker pool pattern solves unbounded concurrency by creating a fixed number of workers that pull tasks from a shared channel. Tasks wait in the queue until a worker becomes free. This bounds the number of concurrently running goroutines, reduces resource pressure, and provides a controlled way to scale throughput horizontally — adding more workers per pod rather than spawning an uncontrolled number of goroutines.
In Go, channels act as pipes connecting goroutines. Workers receive jobs from a channel and send results to another channel or downstream component. The implementation starts with a Worker interface requiring a CompleteJobs function that takes a channel of Job objects. A Job holds the event batch needed for a task. Types like NewProcessorJob extend the base struct for specific use cases, while NewWorker accepts a workFunc callback that processes each job. The workFunc can be adapted per component, so the same Worker interface handles different types of work throughout the pipeline.
The pipeline initializes a job channel and a set number of workers. Each worker calls CompleteJobs in a goroutine, receiving jobs as they become available. As event batches are consumed, they are converted into jobs and sent to the channel with the <- operator.
Graceful shutdown relies on wait groups. A WaitGroup counter is incremented for each worker via producerWg.Add(1) at creation. When the channel closes and no more jobs arrive, CompleteJobs calls wg.Done(), decrementing the counter per worker. On a context cancellation signal (<- ctx.Done()), any remaining batches are pushed to the job channel before it closes. Workers drain the queue, the wait group reaches zero, and the output channel closes safely.
Measured Impact
Load testing showed a direct correlation between worker count and throughput. The original unconstrained design managed about 7.75 thousand events per second per pod in production. Starting with 15 workers each in the processor and producer components delivered 12.9 thousand events per second per pod — a 66% improvement. Increasing workers to 50 brought throughput to 19.3 thousand events per second per pod, a 149% gain over the original. Subsequent optimizations pushed performance to 21 thousand events per second per pod, a 170% increase.
The worker pool design proved its worth during BFCM 2021, when the service handled a peak of 46 thousand events per second. Because the Worker interface is component-agnostic, the same pattern was reused for validation, parsing, and augmentation stages. Each pipeline stage can now be scaled independently with its own worker count to match expected load.
