When a podcast publish took down the site

For 226 episodes of Call Kent, audio processing ran directly on the Fly.io machine that also serves kentcdodds.com. The pipeline was straightforward: stitch caller audio with response audio, trim silence, normalize loudness, add bumpers, and produce the final MP3. It happened inline during the publish request, which was acceptable because only one person ever triggers it.

On March 6, 2026, an extra-long episode finally broke that setup. The primary instance hit extreme CPU saturation, with load average climbing to 400–500% and staying there through the entire FFmpeg run. The CPU quota balance graph showed throttling as the machine exhausted its allocated budget. The site degraded until the job finished, and an emergency upgrade from a shared CPU to a performance CPU was needed to stabilize things.

Fly.io instance metrics from March 6, 2026. Load average spikes to 400–500%. CPU Utilization is saturated. The CPU Quota Balance and Throttling graph shows the machine hitting its CPU quota ceiling.

That incident made the decision clear: FFmpeg had to move off the primary machine.

Why the original design was reasonable

The inline approach wasn't reckless. It was a simple-first choice that survived 226 episodes with minimal incidents. Only the site owner can trigger that code path, and it runs exactly once per episode. Building a full job queue and worker pool from the start would have meant solving a scalability problem that didn't exist yet. "Start simple and iterate when reality tells you to" is a sound approach, and reality finally delivered its signal.

The design also seemed safe because it piggybacked on a machine already sized for general web traffic. The collision only surfaced when the audio was long enough and the shared CPU quota tight enough to create a problem. Waiting also meant Cloudflare Queues and Containers were available when the time came to iterate.

The primary machine was the worst possible host

kentcdodds.com runs on Fly.io with a primary instance handling all writes and read replicas handling reads. FFmpeg running on the primary competed directly with the one machine that couldn't afford to be slow. If the primary stalls or gets throttled, writes stall. Users submitting forms or doing anything stateful hit that bottleneck while replicas stay healthy. The machine that needed to be responsive was the one consuming all the CPU.

Replicas weren't an option—they're read-only, and the publish flow needs to update draft status on the primary. So FFmpeg sat in the worst place possible because it was the only place. The alternative was moving the entire process elsewhere.

The new architecture

The fix moves FFmpeg work entirely off the app server and onto Cloudflare. When an episode is submitted, the app enqueues a job to a Cloudflare Queue containing the draft ID and R2 object keys for both audio files. The app returns immediately instead of blocking on FFmpeg.

A Cloudflare Worker consumes the queue message and forwards the job to a Cloudflare Container. The container pulls audio from R2, runs the FFmpeg pipeline, uploads outputs back to R2, and POSTs a signed callback to the app. The app verifies the signature and advances the draft through its processing states: GENERATING_AUDIO → TRANSCRIBING → GENERATING_METADATA → DONE.

// app/utils/call-kent-audio-processor.server.ts
const res = await fetch(
	`${env.CALL_KENT_AUDIO_CF_API_BASE_URL}/accounts/${env.CLOUDFLARE_ACCOUNT_ID}/queues/${queueId}/messages`,
	{
		method: 'POST',
		headers: {
			Authorization: `Bearer ${env.CLOUDFLARE_API_TOKEN}`,
			'Content-Type': 'application/json',
		},
		body: JSON.stringify({
			content_type: 'json',
			body: { draftId, callAudioKey, responseAudioKey },
		}),
		signal: AbortSignal.timeout(10_000),
	},
)

The admin UI now shows incremental progress labels like "Generating episode audio…" and "Transcribing audio…" instead of hanging on a single blocking request. Transcription and metadata generation still run on the primary app server after the callback—they're candidates for future offloading, but FFmpeg was the immediate problem.

// app/routes/resources/calls/episode-audio-callback.ts
const signature = request.headers.get('X-Signature')
if (!verifyCallKentAudioProcessorCallbackSignature(signature, rawBody)) {
	return new Response('Invalid signature', { status: 401 })
}
const event = parseCallKentAudioProcessorEvent(rawBody)
await handleCallKentAudioProcessorEvent(event)
return Response.json({ ok: true })

The callback signature uses HMAC-SHA256 with a shared secret, verified with a timing-safe comparison to prevent information leakage through response time.

Measured improvement

After the offload was in place, load average on the same primary Fly.io machine peaked around 60–80% during episode processing. No CPU throttling events occurred. Memory stayed stable. The primary machine remained healthy for the entire job duration—roughly an 85% reduction in peak load.

Fly.io instance metrics from March 9, 2026 during the new offloaded FFmpeg run. Load average peaks around 60–80%. Memory is stable. No CPU throttling visible.

The March 6 incident (400–500% load, throttled) versus the March 9 run (60–80% load, stable) shows the difference clearly. Same machine, same app, only the FFmpeg location changed. One caveat: the March 9 episode was considerably shorter, and it ran on the default lite container instance (1/16 vCPU, 256 MiB memory), since upgraded to standard-1 (1/2 vCPU, 4 GiB memory) to give longer episodes more headroom.

What it costs

The cost comparison depends on the alternative. A dedicated Fly.io performance-1x machine runs about $31/month if always on, plus storage and egress. Aggressive auto-stopping cuts that cost but introduces machine lifecycle management and cold start concerns for a publish-blocking job.

Cloudflare's cost shape is usage-based:

  • Cloudflare Queues charges per operation—three per message (write, read, delete). At personal podcast scale, the included 1 million operations/month covers thousands of episodes. Effectively free.
  • Cloudflare Containers bills CPU time, memory, and disk while active. The current configuration uses a standard-1 instance (½ vCPU, 4 GiB memory). A fronting Worker and a Durable Object per container add their own costs.
  • Workers and Durable Objects have separate pricing that matters when estimating total cost—the container isn't the only meter.

For a podcast publishing a few episodes monthly with idle time between runs, Cloudflare scales to zero cleanly; the container costs nothing when asleep. A steady high-volume transcoding workload would change the math, and a dedicated Fly machine might win on simplicity and predictability. The larger benefit wasn't the dollar amount—it was operational isolation. The primary app server is no longer in the blast radius of a long FFmpeg job.

Lessons from the first pass

The architecture direction was right, but the initial implementation had issues that needed cleanup after the PR shipped.

The local fallback had to go. The original PR included a fallback path running FFmpeg locally if the Cloudflare container path failed. That "safety net" was counterproductive: if the container fails and the fallback runs FFmpeg on the primary machine, the outage risk hasn't been reduced—it's just hidden behind a less-frequently-exercised code path. The fallback was removed. The app now throws on enqueue failure and the episode stays in its current state for retry.

Container lifecycle needed explicit management. The first version relied on Cloudflare's built-in sleepAfter timeout to shut down containers after jobs finished. That created two problems:

  1. If an FFmpeg job runs longer than sleepAfter, the container gets reaped mid-job.
  2. If a job finishes in 30 seconds but sleepAfter is 5 minutes, the container sits idle racking up billing on provisioned memory and disk.

The cleaner design being implemented:

  • Keep sleepAfter short as a last-resort backstop (around 1 minute).
  • Have the container send periodic heartbeat pings to the container controller endpoint while FFmpeg runs, renewing the activity lease to prevent premature shutdown.
  • On job completion (in a finally block), send a "finished" signal to a controller endpoint that checks for other active jobs. If none exist, it calls container.stop() immediately.

The container process can't directly tell Cloudflare to stay alive or shut down—that control lives in the Worker and Durable Object wrapper (the container supervisor layer). Heartbeats and stop-if-idle signals must go through that boundary.

The worker was holding the queue open too long. The first version had the queue worker wait for the entire FFmpeg transcode before returning, blocking the worker for the full processing duration. The fix: the container endpoint returns 202 Accepted immediately and runs the job in the background. The worker acks the message and moves on; the container handles the rest asynchronously and sends the callback when done.

Was the migration worth it?

Yes, both because the operational improvement is real and because Cloudflare Queues and Containers proved genuinely interesting to build with. The heartbeat dance is an annoyance that would be better as a built-in feature.

The broader lesson is that simple-first remained the right call. 226 episodes with minimal incidents is a strong record; the original design held up until unlucky timing exposed its limits. When reality finally demanded iteration, the right path was clear and the tooling was available. The takeaway isn't "always use a job queue for compute-heavy tasks"—sometimes the complexity isn't worth the cost until you actually feel the pain.