High Availability by Offloading Work Into the Background - Shopify

Moving Heavy Work Out of the Request Path

An application that spends its time handling slow, resource-intensive, or error-prone work inside the request response cycle risks becoming unresponsive under load. Traffic spikes, slow third-party payment gateways, or lengthy image processing can quickly overwhelm an application server, making it unable to accept new incoming requests. During Black Friday Cyber Monday (BFCM) 2020, Shopify merchants generated sales of over 5 Billion USD, with peak sales exceeding 100 Million USD per hour. At that scale, high availability and short response times are make-or-break. But even for smaller applications, the ability to respond quickly is critical to user experience.

High availability is not the same as high server uptime. A server that hasn't crashed isn't necessarily available; users need to be able to perform meaningful interactions with the application when they need to. If the external-facing portion of the application is also responsible for the heavy lifting of processing requests, it can become a bottleneck, slowing down or blocking new incoming requests. The solution is to offload some of that heavy lifting to a different part of the system, outside the request response cycle, so the application server remains free to accept and serve traffic. This also shortens response times for a better user experience.

Common tasks that are good candidates for offloading include:

• sending emails
• processing images and videos
• firing webhooks or making third party requests
• rebuilding search indexes
• importing large data sets
• cleaning up stale data

The payoff is especially large when the task is slow, consumes significant resources, or is prone to errors.

Consider a user signing up for a web application. The application creates an account and sends a welcome email, but sending the email isn't required for the account to be usable. The user isn't expecting the email to arrive instantly, so there's no reason to make them wait for it within the request response cycle. Similarly, when a user uploads an image, processing the file and generating thumbnails can take a long time, but a successful response isn't necessary for the user to continue interacting with the app. The trade-off is that the server can no longer confirm "the image has been successfully processed." It can only say "the image has been uploaded successfully, it will appear on the website later if things go as planned." Given the significant response time improvement, this is often a worthwhile trade-off.

Background Jobs and Task Queues

Background jobs are a mechanism for offloading work. A background job is a task designated to run later, outside the request response cycle. The application server delegates the task—say, image processing—to a worker process, which can run on an entirely different machine. Since the worker might be busy, the application server can't wait for it synchronously. A message queue between the application server and the worker solves this by making the communication asynchronous. Sender and receiver interact with the queue independently at different times. The application server enqueues a message—the task—and moves on, immediately accepting more incoming requests. The worker processes messages at its own pace. A background job backend is essentially a set of task queues with broker code for managing the workers.

Why Shopify Uses Background Jobs

Shopify queues tens of thousands of jobs per second, leveraging background jobs for several reasons.

Improving Response Times

Background jobs decouple the external-facing request from time-consuming backend tasks. Faster response times for individual requests translate to better overall system availability.

Handling Spikes

A sudden spike in image uploads doesn't hurt if the heavy processing happens off the request path. The application server's availability is bounded by how fast it can enqueue jobs, not by how fast the worker processes them. If the worker falls behind, the queue grows, acting as a buffer. Users can keep uploading images as usual. Shopify has seen traffic spikes of up to 170k requests per second, and background jobs are essential for maintaining availability during such unpredictable surges.

Retries and Redundancy

When a worker hits an error, the job is requeued and retried later. This happens entirely in the background, so it doesn't impact the availability or response times of the application server. This makes background jobs a good fit for error-prone work like requests to unreliable third parties.

Parallelization

Multiple workers can process messages from the same queue, working on several tasks at once. A large task can also be split into smaller subtasks, each queued as an individual background job so they can be processed concurrently.

Prioritization

Most background job backends support prioritization. They may use priority queues that don't follow a strict first-in, first-out order, or set up separate queues for jobs at different priority levels. Workers can process lower-priority jobs when there are no high-priority ones, reducing idle time.

Event- and Time-Based Scheduling

Background jobs aren't always enqueued by the application server. A worker processing a job can queue another job. Jobs can be triggered by events—user interactions or data changes—or by time, such as a daily backup scheduled by a scheduler.

Code Simplicity

The background job backend encapsulates the asynchronous communication between the client requesting the job and the worker executing it. This removes complexity from the job classes themselves, which only implement the task at hand without needing to know about future execution, multiple workers, or retries after errors.

Where Async Gets Complicated

Encapsulating asynchronous communication behind a background job abstraction hides a lot of complexity, but several failure modes remain. A few of them are inherent to the model; others emerge from the way queues and workers are typically deployed.

Job Parameter Drift

The process that queues a job and the worker that executes it do not always run the same version of the code. With canary deployments, that divergence can persist for a while. If a change alters the shape of the job’s parameters, a worker running the older version may try to deserialize a job it no longer understands. Breaking changes to job parameters therefore need a rollout sequence that maintains backward compatibility until every legacy job has drained from the queue.

Delivery Guarantees Are a Choice, Not a Given

When a worker finishes a job, it acknowledges completion and the queue removes the job. But if the worker goes silent mid-job, the queue has a dilemma. Allowing another worker to pick up the job ensures it eventually runs, even if the first worker crashed. However, the first worker might simply have been slow, in which case the job executes twice. Refusing to re-dispatch the job, by contrast, risks it never running at all if the original worker is truly gone.

This is the classic trade-off between at-least-once and at-most-once delivery. The right choice depends on the business impact of duplication. For a charge operation, charging twice may be worse than not charging at all. But if the job is idempotent — checking a tracked state before taking action — a second run is harmless, and at-least-once delivery becomes safe.

The Queue Sits Outside the Transaction

Operational data usually lives in MySQL or PostgreSQL, while the job queue is often backed by Redis. A database transaction writing operational data cannot atomically enqueue a job in Redis. The job is visible to workers immediately, so it may be processed before the enclosing transaction commits — or even if that transaction rolls back.

The typical flow for user input is to write minimal operational data, commit, and then queue a job to perform heavier processing. If the system crashes after the commit but before the enqueue, the job never runs and the system is left inconsistent.

The outbox pattern addresses this by staging job parameters in a table inside the operational database, as part of the same transaction that writes the data. A scheduler periodically scans the staging table, queues the jobs, and marks them as enqueued. Because the update to the staging table can fail after the job was already queued, the job is enqueued at least once and should be idempotent. This approach is sound but adds load to the database and, while it guarantees queuing, it cannot guarantee successful execution.

Coordinating Across Local Transactions

A business process often spans writes from the request-serving application server and from several workers executing jobs. Each write is its own local transaction; eventual consistency is achieved only when the last one commits. If any job fails to commit, the system is inconsistent. The SAGA pattern helps by having jobs report success back to the staging table, which a scheduler checks to detect and correct inconsistencies. That requires even more database work than a transactionally staged queue alone.

Order Is Not Preserved

Jobs leave the queue in a defined order, but once dispatched to different workers, completion order is unpredictable. A failed job that is requeued runs even later. Queueing several jobs in sequence offers no ordering guarantee.

The SAGA pattern can enforce order if the staging table records job sequence. A lighter alternative is to have each job, upon completion, queue the next job in the chain. This requires no staging table or scheduler and avoids extra database load. Its downside is operational: a long chain of jobs queueing jobs becomes hard to trace, and there is little visibility into where a failure occurred.

Long-Running Jobs vs. Frequent Deployments

A job does not need to respond as fast as a user-facing request, but extremely long jobs create operational friction. Resque, for example, refuses to shut down a worker that is mid-job, blocking deployments and making cloud resource usage inefficient. Sidekiq instead aborts and requeues the job, but the retry starts from the beginning. If deployments occur more frequently than the job takes to finish, the job may be aborted repeatedly and never complete. At Shopify’s deployment cadence of about 40 times per day, this is a real concern.

Many long-running jobs have the same shape: iterating over a large dataset. Shopify open-sourced an extension to Ruby on Rails’s Active Job that makes these jobs interruptible and resumable. It sets a checkpoint after each iteration and requeues the job; the next run resumes from the checkpoint. Workers can then be shut down at any time, making them cloud-friendly and deployment-safe. Interruption also enables throttling under database load, or halting jobs entirely to prevent damage, such as during a shard migration.

From Monolith Queues to Distributed Messaging

Ruby backends like Resque and Sidekiq enqueue jobs by serializing an instance of a concrete job class into the queue. Both the enqueuer and the worker must be able to instantiate that class, which works seamlessly in a monolith where both sides run the same codebase. Problems arise when a job should be processed by a separate service, possibly written in a different language.

Sidekiq can be used with separate services, but the workers must remain Ruby and the client must know which Redis queue to target. That constraint makes the approach unwieldy at microservice scale, though it avoids adding a dedicated message broker.

A message broker like RabbitMQ places a purely data-based interface — a JSON payload — between producers and consumers. It can serve as a distributed job backend where workers run entirely independent codebases. Beyond point-to-point queues, topics with routing add flexibility. But messaging has limits: messages are removed after processing, so there is no replay ability and no system-wide source of truth.

Kafka takes a different approach, appending events to a shared, replayable log. The broker is stateless and does not track consumption. Events are organized into topics, giving some publish-subscribe capability for offloading work across services. Because the log persists, it can serve as a source of truth for event sourcing. Append-only writes and a stateless broker yield very high throughput, well suited to real-time data streaming.

Background jobs let the application server shed time-consuming or error-prone work, keeping user-facing latency low and availability high. The queue backend abstracts much of the asynchronous complexity, but the failure modes above require deliberate design — parameter versioning, idempotency, transactional staging, and interruptibility — regardless of whether the queue is a single Redis instance or a distributed event log. By Kerstin Puschke.