During the November 2024 Jake Paul–Mike Tyson fight, Netflix delivered roughly 65 million concurrent streams. On an ordinary day the platform absorbs more than 2 billion requests and upwards of 38 million real-time logging events per second. At that scale no single dashboard or engineering team can hold the whole picture, which makes cross-system investigation the hard part of operating the service.
The MELT foundations and their limits
Metrics, events, logs and traces — MELT — are the raw material for troubleshooting, and each answers roughly a quarter of the question of what caused an incident. Netflix collects all four pillars into a unified observability data layer, covering user-facing experience, application behavior, and backend infrastructure processing.
Each pillar is internally coherent but externally isolated: via Atlas, the telemetry platform, telemetry is tied to applications and time series; Radar holds thresholds and firing history; incident.io holds severities and responders; PagerDuty knows rotations; Spinnaker knows what shipped and when. Slack history stores knowledge no query engine can reach — who solved a similar problem last quarter.
The problem is that these systems disagree about what they are describing. An entity that is an application in one system is a service in a second and a cluster in a third, with names that differ by suffix, casing, or a rename only a few subject-matter experts remember. Joining them has required a human carrying the mapping in their head, which makes that person the bottleneck. The core difficulty is a data-modelling problem rather than a data-volume or dashboarding problem.
The Observability Ontology
What was missing was an explicit, machine-readable agreement about the entities in the operational world. The Observability Ontology supplies it: a formal specification of types, the properties they carry, and the relationships they may have, acting as schema, rule set and shared contract for concepts like applications, services, incidents and deployments. Its unit is the triple, one fact expressed as subject, predicate, object.
api-gateway rdf:type ops:Application .
api-gateway ops:ownedBy "Team Bedrock" .
api-gateway ops:deployedIn "us-east-1" .
INC-5377 rdf:type ops:Incident .
INC-5377 ops:affects api-gateway .
INC-5377 ops:severity "SEV 2" .
Encoding facts this way collapses cross-tool questions — for instance, which incidents affected applications owned by a given team in us-east-1 — into a single query rather than lookups across incidents, Spinnaker and Radar.
The implementation uses the W3C stack: RDF for the data model, OWL for the ontology, SHACL for validation and SPARQL for querying. Because these standards are well specified and widely implemented, the model stays portable across graph stores. Triples are persisted to QuipuDB, an in-house distributed quad store built on redb, the Rust embedded key-value database, generated by parsing raw telemetry, logs and incident discussions. Every harvest also emits a full Turtle serialization for archival, which is human-readable and diff-friendly — a property that has proved more useful for debugging than expected. The ontology is partitioned into twelve namespaces, each owning one operational domain.
Two modelling decisions that mattered
The first was refusing a generic Service type, since "service" carries four different meanings across the tooling. Applications resolve to exactly one type, ops:Application, and each is validated against a software catalog of roughly 49,000 entries before the entity is created. Scraped strings such as usernames, team names and dependency names never become applications. Dependency edges are capped in both directions, because a single popular service can accumulate thousands of reverse dependencies and otherwise dominate the graph.
The second concerned what to model. Incident chat holds the reasoning behind a response — the hypotheses tried and discarded, the mitigation that worked, who knew why — and that knowledge evaporates when the channel goes quiet. A 200-message, 50-person, 15-service incident channel is therefore converted into ordered triples.
Determinism from stochastic tooling
The pipeline leans on inherently stochastic tools: LLMs summarize discussions, fuzzy matching resolves service names, and a probabilistic engine proposes causal chains. None of them is guaranteed to produce identical output twice. The ontology is what turns that pipeline into a deterministic graph that answers a SPARQL query the same way every time.
The pipeline is conceived as a ladder that drives down entropy — the proportion of information still trapped in unstructured form. Constraints on the output space are what make this work: an LLM may say anything, but it cannot emit a triple that violates the type system.
# The model may produce any prose. Its output MUST fit these predicates:
category IN ["incident", "question", "discussion", "decision"]
sentiment IN ["positive", "neutral", "negative", "urgent"]
# Result — a deterministic triple:
thread:123 slack:category "incident" .
thread:123 slack:isIncidentRelated "true"^^xsd:boolean .
The enricher contract
Dozens of enrichers run over the graph. Each takes noisy input, emits typed triples, and reads what earlier enrichers have written:
class Enricher:
async def _enrich_async(self, ...):
raw = await self.fetch(...) # 1. read noisy input
result = self.analyze(raw) # 2. process (may be stochastic)
graph.add((uri, RDF.type, OPS.X)) # 3. emit TYPED triples
graph.add((uri, OPS.pred, Literal(result.value)))
return EnrichmentResult(success=True, triples_added=count)
Three invariants hold across all of them: every entity carries an rdf:type, every URI follows a single naming scheme, and the catalog gates every ops:Application. The enrichment pipeline runs sequentially — load the graph, add triples, save — which eliminates a class of race conditions at a wall-clock cost the team accepts. Calls to external systems such as metrics, logs and events are best-effort, so a slow downstream API degrades rather than blocks a harvest. Work is also underway to express those systems as their own ontologies, such as a Metrics Ontology and an Alerts Ontology, so everything can be queried from the ontology itself.
Correlation with evidence attached
The correlation engine condenses signals from diverse sources into causal or correlatory chains. It currently reads thirteen signal types, among them log errors, metric anomalies, slow and failed traces, alerts, incidents, deployments, config changes, experiments, autoscaling, commits, end-to-end bottlenecks and error patterns, then searches for paths among them. Some ordered pairs are known-causal and carry a high prior weight — deployment before log error, config change before metric anomaly, failed trace before incident. Others exhibiting high correlation become correlation chains. Chains are scored on temporal proximity and severity within a ten-minute window, and anything below 0.6 confidence is discarded.
The engine proposes; it does not conclude. It stores its chains together with the graph records used to derive them, so a 0.87 chain is a strong hypothesis a human or agent can verify, with the graph recording the confidence alongside the claim.
The knowledge flywheel
Each harvest cycle repeats the same four steps, and each rotation leaves the graph richer: collect the signals, add typed triples, find patterns invisible in the raw data, then feed knowledge back so established patterns become detection rules. A causal chain discovered in one incident becomes a known pattern for the next, and the ontology gradually stops being a schema and starts being a memory.
Once an incident in api-gateway caused by a timeout is resolved, that knowledge is converted into ontology entries. Over repeated cycles the graph learns patterns until they become wisdom of the form "api-gateway deploys cause 40% of SEV-2s" and "30% of deployment issues in api-gateway come from Connection Timeouts." Surfaced as insights, these let teams reduce incidents proactively rather than only react to new ones.
AutoRCA and the ontology as interface
AutoRCA is the internal agentic AI solution built on top of the graph, using a set of specialist agents coordinated around a single question. Working the home-screen regression from the start of this piece, the reasoning chain runs roughly as follows:
Each specialization is built by engineers with deep domain knowledge, and that division of labor only works if specialists can pass findings to one another without agreeing on communication terms or even knowing the others exist. The ontology is the interface: each specialist learns one vocabulary, the graph's. A team writing trace use cases records findings as triples about an ops:Application; the logs specialization reads triples about the same ops:Application; neither needs to know about the other. A new specialist learns the vocabulary once and can immediately read everything the other six have written.
The change is narrower than it sounds: an alert still fires, an engineer still weighs the evidence, and a fix is still written by someone who understands the code. What the ontology absorbs is the hour or two previously spent assembling context — finding the owning team, reconstructing what shipped, correlating evidence from different parts of the system, recalling a similar incident from last quarter. Ranked hypotheses are still sometimes wrong, which is why each carries the evidence that produced it, but the investigation now starts with the picture that used to take half an afternoon to build.
Principles for a second attempt
- Model identity first. Choose one authoritative catalog per entity kind and validate against it before creating an entity.
- Treat the ontology as code. Version, review and test it; let changes arrive as pull requests. This supports agentic development and lets teams scale contributions.
- Separate concepts from instances. Concepts change on a human timescale, instances change continuously; conflating them makes both harder to evolve.
- Build on open standards. RDF, OWL, SHACL and SPARQL are specified, multiply implemented and portable; a bespoke graph model ties future consumers to one implementation.
- Constrain stochastic tools with a type system. The principle the team would defend hardest: models are useful exactly where determinism is impossible, and a sufficiently narrow output contract turns their output into something a query engine can trust.
- Make enrichment best-effort and ordered. A slow dependency should degrade the graph, never block it, and ordering enrichers buys accuracy at the cost of parallelism.
- Model human reasoning alongside machine telemetry. The most valuable edges came from discussion. Telemetry records what the system did; discussion records what people understood.
- Keep humans as approvers. Agents may propose fixes, enrichers and ontology changes, but every proposal goes through human review.
Where the graph is heading
Three capabilities follow from the graph: automatic root cause analysis, where a regression arrives with a ranked set of graph-backed hypotheses; automated remediation, where a sufficiently confident diagnosis carries a proposed action such as a targeted rollback, a traffic shift or a code change; and eventually infrastructure that heals itself by acting on a predicted failure before users feel it.
In parallel, the team plans to integrate the Observability Ontology with Upper, the domain-modelling language at the heart of Netflix's Unified Data Architecture (UDA). Upper lets teams describe business and system domains as interconnected machine-readable models and preserve meaning and constraints as those models are projected into concrete schemas. The integration would connect operational concepts such as applications, alerts, deployments and incidents to domains modeled across Netflix, giving engineers and agents a shared semantic layer for querying, validation and reasoning. It starts with a proof of concept modelling incidents in Upper.
