From rule-based classification to AI-assisted extraction
Netflix's data and ML practitioners run millions of workloads daily across Titus, Spark, Trino and Druid, and on peak days roughly 100,000 of those workloads fail. Locating the useful signal among several logs, deciding whether the cause is user code or the platform, and choosing between retry, reconfiguration and escalation all consume engineering time.
The first response was Pensive, an auto-diagnosis and remediation system that pulled logs from failed workloads, applied regular expressions and hand-authored Java rules to classify errors, and passed retry or remediation guidance to products such as Maestro. It made about 50% of failures self-service and in some cases automatic. Rules only extend to failure modes someone has recognized and encoded, and they need upkeep as engines and log formats change: across one three-month period Pensive processed roughly seven million job-failure classifications, yet about half landed in the unknown bucket.
Pensive AI adds AI-assisted error extraction plus focused diagnosis and remediation guidance. The design wraps AI around deterministic processes rather than placing a model in front of a log corpus, which yields broader coverage, low latency, controlled AI costs and easier maintenance. AI error extraction now covers 100% of failures, including those legacy Pensive classified as unknown. Failed step attempts with no Pensive AI usage averaged 321% more manual debug-log exploration actions than attempts where it was used — an observational comparison, not a causal claim.
The extraction path: spans instead of transcripts
Legacy rules are precise but each captures one log shape:
- name: SparkContainerOOM
logSource: GENIE_STDERR
restartable: false
regex: "Container.*is running beyond physical memory limits.*Killing container"
summary: Spark AM container runs out of memory
Pensive AI extracts errors that have no matching rule and adapts when a familiar error changes shape after a platform upgrade. Data Portal, the UI for data pipelines, surfaces extracted errors when available and links back to source logs.
Because generating output tokens is far slower than reading input tokens, asking a model to transcribe a long stack trace would be costly and slow. Pensive instead numbers each log's lines, supplies a sequence of {lineNumber, text} records, and asks the model for the first and last line of the error span. Pensive reconstructs the text from its own copy, so the model always returns two integers regardless of span length, the displayed content comes from the original log, and hallucination is reduced. The model can still pick the wrong range, which is why extraction quality stays a core eval measurement.
For orchestrated workflows, "Pensive AI Error Extraction" runs automatically on every failed step attempt — one execution of a step within a Maestro workflow. Maestro already calls Pensive for a regex-derived error and restartability classification, so AI extraction rides the same request and returns alongside the established fields. The whole Pensive response must fit Maestro's hard 90-second post-processing timeout.
Logs are therefore extracted in parallel, and the per-log calls share a single deadline set well inside Maestro's limit; a call that misses it is dropped from the result so the overall request still returns in time. Latency for extraction is a few seconds, versus tens of seconds or a minute for full debugging.
The data contract also needed care. The legacy model held one flattened error string; AI extraction returns a list of errors, one per log source, with deep links to the originating logs. Rather than changing the old field in place and breaking clients, Pensive introduced the new model beside the legacy non-AI one and continued producing both. Data Portal preferred AI errors when present and fell back otherwise, allowing gradual rollout; historical failures still show an error because no backfill was done, given log retention limits. Today the Data Portal surface for failed Maestro steps reads the AI field, while the query editor, Workbench notebooks and some Slack clients still use the legacy path, each needing a compatible schema change before adopting the new data.
On-demand debugging, cached per attempt
Pensive AI Debugging analyzes failed step attempts and returns root-cause analysis with suggested remediations. It is reachable through programmatic APIs, MCP and the Data Portal UI, all behind a common entry point across the Netflix Data Platform. Debugging runs on demand rather than automatically, since most failed steps are never opened, which keeps costs down.
The unit of work is the failed step attempt rather than the person who triggered it, so everyone investigating it sees the same diagnosis — and that gives a stable cache key. Responses are cached by step attempt, turning a second request into a key-value read instead of another inference call, and a debugging generated in Data Portal is served straight to the MCP path from cache. A conditional write creates an in-progress record: one caller generates, others wait on the shared result. Records still in progress past an abandonment timeout are reclaimed; completed responses expire after 90 days.
A caller can read a cached response, generate one if none exists, or regenerate and overwrite. Those semantics serve the UI, programmatic access and the evaluation harness while keeping each response tied to one immutable workload failure.
Responses stream as they are produced: the service emits server-sent events and the GraphQL layer republishes them as a WebSocket subscription for Data Portal. Each event carries the complete response so far — slightly duplicative, but it removes the need for a replay buffer or token reassembly, and the next event corrects a dropped or late one. Generation is decoupled from the requesting connection, so a disconnect does not cancel work and the final answer still lands in the cache; a hard reload can poll for the completed result. Within the Data Portal single-page application, active streams outlive view and component lifecycle, so navigation doesn't interrupt a response.
Evaluation built on frozen rules and archived logs
Separate frameworks cover the two jobs. For extraction, legacy hand-written rules supply reference labels: running them over production logs and comparing their spans with Pensive AI output gives a regression set without manual relabeling, though it is not absolute ground truth since the rules can go stale. They are kept frozen as a label source.
Repeatable experiments also need stable inputs. Logs expire after as few as seven or as many as 30 days, so production requests archive fetched logs to S3 asynchronously during workload execution, off the user-facing critical path. Uploads never overwrite an existing object, preserving the exact snapshot behind an evaluation case; scores then change only with the system under test. The archive shields source systems from parallel experiments and cached S3 inputs cut eval costs.
Four signals are measured for extraction:
- Inclusion: an LLM judge checks whether the legacy error is still represented in the extracted errors, even where wording differs.
- Precision: a second judge sees only the extracted errors and assesses whether each is meaningful and actionable alone.
- Repeated noise: a script flags lines recurring across the dataset. Lines appearing in more than about 30% of workflows become exclusion candidates; the list ended up at three lines, each otherwise present in nearly every extracted error.
- Latency: every extraction call's latency is recorded, since quality gains that cost too much work often track rising AI costs.
Together these close the loop: prompts can remove general noise, the dataset view exposes boilerplate prompts miss, rules act as a regression floor, and archived logs make comparisons repeatable.
Debugging ground truth is seeded by mining internal engineering-support Slack threads; an LLM identifies threads linking to an orchestrated workflow, describing a resolved problem that matches the class of issue Pensive AI handles, and a second model synthesizes a reference response from the discussion. Paired with that failure's logs, it becomes an evaluation row. This works only retrospectively, since in production Pensive AI aims to guide debugging before such a thread exists. Tracing on a Braintrust-based backend puts prompt, completion, tool calls and latency in one view, with child spans per log fetch and extraction tool call; the trace identifier is returned with API and UI responses so feedback links directly to the trace.
Gating releases and reading user signals
Log selection, trimming logic and source-system changes alter what the model sees, so the same harness and evaluators run from local development through deployment. On each pull request the branch deploys to an isolated preview environment and a CI job runs an eval experiment against it, with precision and inclusion thresholds gating the merge. The job also runs after a build is promoted to catch interactions absent when individual PRs were evaluated in isolation; a missed threshold alerts the on-call engineer to investigate or roll back. A scheduled run against production surfaces gradual drift, and developers can trigger experiments remotely along the same deployed-environment path, keeping results comparable everywhere.
Manual debug-log exploration is the main behavioral signal, available from product telemetry and closest to what the feature targets: Pensive already links to logs for a failed step attempt, and comparing attempts with and without Pensive AI interaction produced the 321% figure. It does not establish causality or a specific reduction in mean time to resolution, but suggests focused diagnosis means inspecting fewer raw logs. Explicit thumbs up and thumbs down land beside the internal traces, mapping feedback to full inputs, outputs and the path taken; a business-hours alert sends each thumbs down to Slack, which helped resolve early rough edges and context gaps and still feeds continuous improvement.
What comes next
Pensive AI Deepthink is an opt-in managed agentic experience under exploration, with higher cost and latency. It builds on Debugging's precise, low-latency analysis and gathers more context from internal documentation, manuals, past Slack discussions, source code, read-only warehouse queries and Spark execution details. A routing step sends the investigation to a skill such as Spark failures, data-audit failures or late workflows, keeping the shared starting point deterministic while the investigation specializes to improve recall. The team is also in talks with the Spark team about using Pensive AI responses for failure debugging analysis and A/B comparison ahead of upcoming platform migrations.
Pensive AI Error Classification is planned to take over restartability classification and the platform-versus-user split that hard-to-maintain legacy rules still handle; Maestro acts on those labels, failing fast or adjusting default retries, with platform errors retried more than user errors by default. Classifications need a higher bar than extraction or debugging alone. Moving most of them to AI would retire much of the remaining rule maintenance and avoid retrying unknown failures unlikely to succeed. The costliest mistake is calling a recoverable failure non-restartable, which turns a recoverable step into a hard failure, alerts users and forces manual remediation — so the evaluation framework must be tuned around that outcome before AI takes such decisions, while avoiding retries of non-transient failures that waste compute and delay pipelines.
Reliability here comes from the system built around model calls: bounded extractions, compatible client contracts, responsive delivery, reproducible evals, caching and tracing, and release controls that keep quality and cost visible as the platform changes.
