Transactionally Staged Job Drains in Postgres

Why Queued Jobs Fail Right After Commit

Background jobs are a standard way to keep user requests fast by moving slow work — API calls, email delivery, data enrichment — out of band. But when jobs are enqueued from inside a database transaction, a subtle race condition appears: the job can be picked up by a worker before the transaction that created its data has committed.

Consider a transaction that inserts a user record and then queues a job that needs to read that record. If the worker runs immediately, the record isn’t visible yet. Even if the job is retried a few times, the specific timing of the commit and the worker’s retry schedule can cause repeated failures before the data finally appears. The job isn’t broken — it’s just too early.

Rollbacks present the opposite problem. If the transaction is discarded after the job was enqueued, the job references data that no longer exists. It will fail permanently, no matter how often it’s retried.

Popular job systems acknowledge this. Sidekiq’s FAQ, for example, recommends using Rails’ after_commit hook or moving job creation outside the transaction. But both workarounds have their own flaw: if you queue the job only after commit, a process crash in between leaves the data committed with no job ever queued. That failure is silent and far harder to detect.

Staging Jobs in the Database

A cleaner solution is to stage jobs inside the database itself, using the transaction’s own ACID guarantees as a gate. Jobs are inserted into a staged_jobs table rather than sent directly to the queue. A separate enqueuer process polls that table, forwards any visible rows to the real job queue, and then deletes them.

A minimal table definition looks like this:

CREATE TABLE staged_jobs (
    id       BIGSERIAL PRIMARY KEY,
    job_name TEXT      NOT NULL,
    job_args JSONB     NOT NULL
);

The enqueuer implementation, sending jobs to Sidekiq, is straightforward:

# Only one enqueuer should be running at any given time.
acquire_lock(:enqueuer) do

  loop do
    # Need at least repeatable read isolation level so that our DELETE after
    # enqueueing will see the same jobs as the original SELECT.
    DB.transaction(isolation_level: :repeatable_read) do
      jobs = StagedJob.order(:id).limit(BATCH_SIZE)

      unless jobs.empty?
        jobs.each do |job|
          Sidekiq.enqueue(job.job_name, *job.job_args)
        end

        StagedJob.where(Sequel.lit("id <= ?", jobs.last.id)).delete
      end
    end

    # If `staged_jobs` was empty, sleep for some time so
    # we're not continuously hammering the database with
    # no-ops.
    sleep_with_exponential_backoff
  end

end

Because the enqueuer only sees committed rows, a job staged by an uncommitted transaction is invisible until that transaction commits. If the transaction rolls back, the staged row disappears along with all other changes. There’s no window where a worker can run too early, and no orphaned work after a rollback.

This design also gives at-least-once delivery. Rows are deleted only after the enqueuer has successfully transmitted the job to the queue. If the enqueuer crashes mid-batch, it simply picks up the remaining rows on its next run — no job is lost.

Why Not Just Use an In-Database Queue?

Tools like Delayed Job, Que, and Queue Classic already use transactional mechanics to keep uncommitted jobs hidden, with workers pulling jobs directly from a database table. That works at modest scale, but it degrades under load: workers aggressively competing to lock rows create churn that a busy database doesn’t handle well. Postgres in particular suffers from long-running transactions, which increase the time workers need to claim a job and can push the queue into a spiral.

The staged drain sidesteps that contention by selecting ready jobs in bulk and handoff to a store like Redis, which is built for distributing work to many competing workers. The database does what it’s good at — transactional consistency — without becoming the bottleneck for job distribution.

A Working Example

Since the pattern gained traction, a ready-made implementation exists for Sidekiq: the sidekiq-staged_push gem, which brings transactionally staged job pushes to that ecosystem.