Video podcast publishing delays on June 24
For several hours on June 24, video podcast episodes were delayed in publishing to Spotify. The incident was caused by video transcoding infrastructure reaching maximum capacity. Episodes that normally publish within minutes took hours, and creators who re-uploaded episodes that had not appeared added further load to the system. The system failed to confirm that uploads were received and queued, which compounded the problem.
Why the pipeline stalled
Four factors combined to create the backlog:
- Insufficient capacity headroom. Transcoding infrastructure could handle typical submission volumes and scale alongside low-priority work, but lacked headroom for large spikes in content delivery.
- A scheduled batch job. Re-processing of existing episodes for compatibility with playback system changes was consuming capacity. It appeared harmless earlier in the day but became problematic when combined with increased submissions.
- Higher per-item processing cost. A recent change to video transcoding improved quality at lower bitrates, but increased the processing time and power each episode requires. This added demand was not fully accounted for in capacity planning.
- A resource scheduling bug. After migration to more powerful hardware, a bug left available processing capacity underutilized, cutting throughput by about 10%.
Response and timeline
Once the issue was recognized, the batch job was stopped, the scheduling bug was fixed, and additional processing capacity was added overnight. All backlogs cleared by the following morning, and publishing returned to normal.
- 13:30 — Early alerts fire in internal monitoring; not yet recognized as a broader capacity issue.
- 15:00 — Video podcast delivery spike pushes transcoding close to maximum capacity.
- 16:35 — Batch processing job stopped to free capacity.
- 17:31 — First creator report of an issue impacting podcast video publishing received.
- 17:34 — Automated alerts confirm queue backlog exceeding thresholds; incident response begins.
- 19:00 — Creator reports escalated to incident team.
- 20:49 — Software fix deployed to improve resource utilization.
- 00:14 (Jun 25) — Additional processing cluster brought online.
- 01:02 — All queues cleared.
- 07:30 — Full confirmation: all publishing pipelines operating normally.
Roughly four hours elapsed between the first alerts and formal incident response. Engineers investigating the early alerts stopped the batch job at 16:35, but the full scope of the capacity problem was not recognized until queues breached thresholds at 17:34. Monitoring improvements aim to close that gap.
Remediation and reliability program
Immediate corrective steps included increasing transcoding capacity by approximately 67%, fixing the resource scheduling bug, and improving monitoring to alert earlier when capacity approaches limits.
A dedicated cross-team effort is now focused on broader reliability improvements to the publishing pipeline:
- Capacity planning that accounts for burst capacity and incident recovery, not just steady-state traffic.
- Prioritization across publishing systems so real-time content from creators is always processed ahead of background operations.
- Rate limiting and backpressure mechanisms throughout the pipeline to handle unexpected load gracefully.
- Improving notification processes so creators are informed as soon as possible when things aren't working.
