Shining a Light on Build Failures
Every Cloudflare Workers Builds project is built from source control with minimal configuration, but when a build fails, it's not always obvious whether the problem is something in the build pipeline or a mistake in the user's project. For a long time, the team would only find out about systemic issues when customers opened support tickets.
The platform itself is built end-to-end on Cloudflare's own developer stack: Workers, Durable Objects, Hyperdrive, Containers, Queues, KV, and R2. The goal of Workers Builds is to keep that stack reliable, and that means catching issues before customers do. The first step was a clear definition of what counts as a successful build:
- The build is started successfully.
- Attempting to install tools and dependencies succeeds.
- The user-defined build/deploy commands run successfully.
- The build is marked stopped in the database.
Failures in steps 1 or 4 are the platform's responsibility. Failures in steps 2 or 3 get classified as user error — but that designation often hid real problems. A significant number of builds were failing because of something in the build environment or documentation that pushed developers to misconfigure their commands. Distinguishing actual user mistakes from product gaps required reading the logs.
From Manual Log Checks to Automated Classification
Looking at raw logs one by one was quick to spot-check but impossible at scale. The team needed automation that could pull logs from a failed build, classify the error, and keep aggregate results free of account IDs and customer-specific paths.
BuildBuddy, the Durable Object that manages each build's lifecycle, already receives and stores build logs. A new Worker Queue called BuildErrorsQueue was added as a post-failure pipeline: once a build fails, BuildBuddy sends the build's ID to the queue. The consumer fetches the logs, runs matcher patterns against them, and saves the results to Postgres.
Static pattern matching was the first iteration:
export const DetectedErrorCodes = {
wrangler_error: {
detect: async (lines: LogLines) => {
const errors: DetectedError[] = []
for (const line of lines) {
if (line[2].trim().startsWith('✘ [ERROR]')) {
errors.push({
error_code: 'wrangler_error',
error_group: getWranglerLogGroupFromLogLine(line, wranglerRegexMatchers),
detected_on: new Date(),
lines_matched: [line],
})
}
}
return errors
},
},
installing_tools_or_dependencies_failed: { ... },
}That catches errors but lumps every Wrangler issue into a single bucket. To get more granular, log lines are normalized into groups so the same underlying failure shows up consistently:
function getWranglerLogGroupFromLogLine(
logLine: LogLine,
regexMatchers: RegexMatcher[]
): string {
const original = logLine[2].trim().replaceAll(/[\t\n\r]+/g, ' ')
let message = original
let group = original
for (const { mustMatch, patterns, stopOnMatch, name, useNameAsGroup } of regexMatchers) {
if (mustMatch !== undefined) {
const matched = matchLineToRegexes(message, mustMatch)
if (!matched) continue
}
if (patterns) {
for (const [pattern, mask] of patterns) {
message = message.replaceAll(pattern, mask)
}
}
if (useNameAsGroup === true) {
group = name
} else {
group = message
}
if (Boolean(stopOnMatch) && message !== original) break
}
return group
}
const wranglerRegexMatchers: RegexMatcher[] = [
{
name: 'could_not_resolve',
// ✘ [ERROR] Could not resolve "./balance"
// ✘ [ERROR] Could not resolve "node:string_decoder" (originally "string_decoder/")
mustMatch: [/^✘ \[ERROR\] Could not resolve "[@\w :/\\.-]*"/i],
stopOnMatch: true,
patterns: [
[/(?<=^✘ \[ERROR\] Could not resolve ")[@\w :/\\.-]*(?=")/gi, '<MODULE>'],
[/(?<=\(originally ")[@\w :/\\.-]*(?=")/gi, '<MODULE>'],
],
},
{
name: 'no_matching_export_for_import',
// ✘ [ERROR] No matching export in "src/db/schemas/index.ts" for import "someCoolTable"
mustMatch: [/^✘ \[ERROR\] No matching export in "/i],
stopOnMatch: true,
patterns: [
[/(?<=^✘ \[ERROR\] No matching export in ")[@~\w:/\\.-]*(?=")/gi, '<MODULE>'],
[/(?<=" for import ")[\w-]*(?=")/gi, '<IMPORT>'],
],
},
// ...many more added over time
]With matchers and normalizers in place, the queue consumer implementation is straightforward — fetching logs per build, classifying the failure, and writing results back to Postgres while clearing any prior entries to keep the dataset clean as the detection patterns are refined:
export async function handleQueue(
batch: MessageBatch,
env: Bindings,
ctx: ExecutionContext
): Promise<void> {
...
await pMap(batch.messages, async (msg) => {
try {
const { build_id } = BuildErrorsQueueMessageBody.parse(msg.body)
await store.buildErrors.deleteErrorsByBuildId({ build_id })
const bb = getBuildBuddy(env, build_id)
const errors: DetectedError[] = []
let cursor: LogsCursor | undefined
let hasMore = false
do {
using maybeNewLogs = await bb.getLogs(cursor, false)
const newLogs = LogsWithCursor.parse(maybeNewLogs)
cursor = newLogs.cursor
const newErrors = await detectErrorsInLogLines(newLogs.lines)
errors.push(...newErrors)
hasMore = Boolean(cursor) && newLogs.lines.length > 0
} while (hasMore)
if (errors.length > 0) {
await store.buildErrors.insertErrors(
errors.map((e) => ({
build_id,
error_code: e.error_code,
error_group: e.error_group,
}))
)
}
msg.ack()
} catch (e) {
msg.retry()
sentry.captureException(e)
}
})
}Backfilling More Than a Million Historical Builds
Applying error detection to new builds alone meant waiting days for enough failures to accumulate in order to tune the matchers. The real need was to run the whole detection pipeline over the backlog — over one million failed builds stored across one million-plus Durable Objects — without waiting weeks for a manual sweep.
A Kubernetes-style long-running worker would handle this, but Workers wouldn't: each Durable Object alarm invocation has a limited work window. That's the key constraint for the backfill design. Instead of building one massive job runner, the team pushed the historical builds through the BuildErrorsQueue concurrently, so the same infrastructure the platform runs on does the heavy lifting with no long-running process to keep alive.
The piece that coordinates the backfill is a Durable Object class — BuildErrorsAgent, a single instance that pulls a batch of build IDs from Postgres and enqueues them for detection. Backfill parameters are persisted in Durable Object storage:
async start({
min_build_id,
max_build_id,
}: {
min_build_id: BuildRecord['build_id']
max_build_id: BuildRecord['build_id']
}): Promise<void> {
logger.setTags({ handler: 'start', environment: this.env.ENVIRONMENT })
try {
if (min_build_id < 0) throw new Error('min_build_id cannot be negative')
if (max_build_id < min_build_id) {
throw new Error('max_build_id cannot be less than min_build_id')
}
const [started_on, stopped_on] = await Promise.all([
this.kv.get('started_on'),
this.kv.get('stopped_on'),
])
await match({ started_on, stopped_on })
.with({ started_on: P.not(null), stopped_on: P.nullish }, () => {
throw new Error('BuildErrorsAgent is already running')
})
.otherwise(async () => {
// delete all existing data and start queueing failed builds
await this.state.storage.deleteAlarm()
await this.state.storage.deleteAll()
this.kv.put('started_on', new Date())
this.kv.put('config', { min_build_id, max_build_id })
void this.state.storage.setAlarm(this.getNextAlarmDate())
})
} catch (e) {
this.sentry.captureException(e)
throw e
}
}The alarm does the recurring work. Each invocation follows the same order:
- Schedule the next alarm first, so a failure can't stop the job dead.
- Read the job state from storage.
- Confirm the backfill is still configured to run and hasn't passed its max build ID.
- Query Postgres for the next batch of builds and enqueue them for error classification.
Pattern matching with ts-pattern made the state transitions in the alarm much more readable than procedural alternatives. The more powerful XState library was considered but dropped in favor of the simpler utility. This approach matches how the team thinks about Workers: a single Durable Object instance acting as a coordinator and delegating work to queues means one API call can process over a million builds in about two hours.
The first backfill run classified about 80 percent of historical failures with reasonable error codes. Because the backfill is re-runnable in minutes, tuning the matchers and running again is an iterative, daily process instead of a project-by-project one.
Customer-Facing Results Already Landed
The aggregate error reports are starting to drive fixes in the product, not just in the control plane:
- Wrangler now shows a distinct message when no config file is found, instead of a cryptic failure.
- Several edge cases where the wrong package manager was selected for TypeScript or JavaScript projects are fixed.
- Support for
bun.lockwas added; onlybun.lockbwas recognized previously. - Monorepo build caching works again in scenarios that previously broke it.
- Python projects specifying a version in
runtime.txtno longer fail on builds.
The error detection core — the queue plus the coordinating Durable Object — came together in about two days of engineering time. Living up to the "Customer Zero" philosophy means that Workers Builds itself is the first user of the platform's own primitives, catching its own problems the same way any customer's project would.
Beyond the fix: tooling and next steps
The build detection work is part of a broader push to make Workers development smoother. Beyond reliability and speed improvements, Cloudflare has begun exploring how to surface the same issues that the detection system catches in more developer-friendly ways.
One concrete output is the Builds MCP server, an open-source tool that lets developers debug Workers Builds directly from AI-assisted editors like Cursor or Claude. Instead of digging through raw logs, users can query build status and errors in their normal development environment.
There is also ongoing work to expose detected issues in the Cloudflare Dashboard. The goal is to let users identify problems at a glance, rather than paging through hundreds of log lines to find the root cause.
Try it yourself
If you want to see Durable Objects and Workers Builds in action, Cloudflare provides a ready-made chat application template. It’s a straightforward way to test the deployment pipeline and the build detection logic covered here.
