From One Crontab to a Distributed Scheduler
Slack’s cron scripts handle a surprising amount of critical work: firing reminders on time, dispatching email notifications, and cleaning up databases. As the number of scripts and the volume of data they touch grew, so did the pain of maintaining the original setup — a single node running every script from one crontab file. Scaling meant migrating to bigger nodes with more CPU and RAM, which worked for a while but left the system fragile. Any hiccup in provisioning, rotation, or configuration on that one box took down key Slack functionality.
The team decided the patchwork had run its course and built a dedicated cron execution service. Rather than reinventing every piece, the new architecture leans on existing internal infrastructure wherever possible: a Golang scheduling service running on Bedrock, Slack’s Kubernetes wrapper; Slack’s Job Queue for the heavy lifting of executing scripts; and a Vitess table for deduplication and monitoring visibility.
Scheduling with a Single Active Pod
The heart of the new system is the “Scheduled Job Conductor,” a Golang service that replicates cron behavior using a Golang cron library. Keeping the same cron string format used on the old box meant migrations were straightforward and less error-prone. The service runs on Bedrock, which makes spinning up multiple pods trivial. But only one pod actively schedules at any given time; Kubernetes Leader Election designates the active scheduler while the others remain in standby, ready to take over quickly if the leader fails.
Why not spread scheduling work across all pods? Synchronizing multiple schedulers would be more trouble than it’s worth. Leader switches happen fast enough that downtime is unlikely, and the real CPU- and memory-intensive work — actually running the scripts — gets offloaded to the Job Queue. That leaves the scheduling pod itself lightweight, with the other pods acting purely as insurance.
One pragmatic detail: the system tries to avoid letting the leader go down at the top of a minute, since that is precisely when cron scripts are most likely to be due. This eases transitions and reduces the chance of missed schedules during failover.
Offloading Execution to the Job Queue
Actual script execution is handled by Slack’s Job Queue, an asynchronous compute platform that already processes roughly 9 billion jobs per day. Jobs flow through logical queues: first into Kafka for durable storage in case of failure or backlog, then into Redis for short-term storage enriched with metadata about which worker is executing the job, and finally to a job worker node that runs the code.
In this context, one job equals one cron script. Although the Job Queue is asynchronous, it executes work quickly when a job is isolated on its own queue — precisely how this system taps into it. Offloading script execution to an existing platform removed the compute and memory scaling concerns entirely, and since the Job Queue is already critical to Slack’s operations, maintenance effort stays minimal.
Tracking Runs in Vitess
The third component is a Vitess table that serves two purposes: deduplication and user-facing monitoring. The old cron system relied on flock, a Linux locking utility, to prevent two copies of a script from running simultaneously. That worked for most scripts, but any script whose runtime exceeded its recurrence interval could spawn overlapping runs. In the new system, every job execution is recorded as a new row, with its state updated as it moves through the pipeline — from enqueued to in progress to done. Before launching a new run, the system queries for any active job of that script name, using an index to keep lookups fast.
That same state data powers a simple internal web page where engineers can check the status of their script runs and inspect any errors. This visibility matters because some scripts take up to an hour to finish; users need to confirm their work is still progressing rather than silently failing.
A More Durable Foundation
The single-crontab approach served Slack for years but eventually became a bottleneck rather than a convenience. By splitting scheduling from execution and adding persistent tracking, the new service offers a more reliable and scalable path forward — one that should give Slack room to grow without the same operational fragility.
