Why GPU health needs its own reliability layer
A large training job runs as a gang: hundreds of GPUs spread across many machines, ranks stepping in lockstep and exchanging data every step. The job moves at the pace of its slowest participant and stops outright if one dies. Every GPU, high-speed network adapter, and interconnect must hold for the entire run — which can last weeks. At modern scale, individual devices and links will fail mid-run, so the platform has to catch and handle that without a human.
Netflix's personalization models are the visible case: rows, ordering, titles, and artwork on the home screen are chosen by ML trained on viewing behavior, and a growing share of that infrastructure is GPU-based for both training and increasingly for live serving. GPU capacity is also one of the fastest-growing and most expensive parts of the fleet, which puts reliability on two fronts:
- Availability — a training run that loses a machine, or an inference service that cannot get healthy capacity, delays work members and creators depend on.
- Cost efficiency — an idle GPU behind a slow neighbor, or a whole machine pulled from the pool because one device degraded, is paid-for capacity doing nothing. More reliability means more useful work per GPU.
As more AI features reach members, inference takes a larger share of that footprint, so the properties that matter for multi-week training runs now matter for always-on serving too. These workloads run on our Kubernetes-native Compute platform.
What instance-level and driver-level checks miss
We build on the cloud provider baseline: a system status check for host power, network, and hardware reachability; an instance status check for OS liveness; a check on attached storage; scheduled maintenance events (which also feed one of our own checks); and automatic instance recovery. All of these are in use. They catch a machine that is down, unreachable, or slated for retirement, and they stop at the instance boundary — confirming the box is up and reachable, not that each GPU, its memory, its interconnect, and the fabric can support distributed training.
A driver-responds check that confirms GPUs are visible is the usual next step. It catches a dead or missing device but misses the failures that cost the most time:
- A GPU can answer a status query and still be faulty once a workload touches it. Faults in on-chip memory or device firmware stay invisible until a job allocates memory or runs sustained compute, then the job fails or crawls.
- A high-speed network adapter can be present with fewer links than expected. Collectives still run at a fraction of the bandwidth, so the job does not crash — it slows, and the cause is easy to overlook.
- Checks can fail into looking healthy. A diagnostic tool hangs when the driver hangs, and a naive check waiting on it hangs too. A binary the check depends on can disappear in a driver upgrade, and the check silently stops running.
Any of these on any one machine degrades or stops the whole gang, and none appears in an instance-level or driver-responds check. The health system must assume hardware can lie and inspection tools can break.
The four layers
Reliability is treated as four layers, each catching what the layer below cannot:
1/ Node and hardware health: find bad GPUs, network adapters, interconnects, memory errors, thermal issues; taint or recycle the machine.
2/ Scheduler and platform: retry at task and job level, validate a machine before it takes work, drain and recycle safely.
3/ Training loop: checkpointing plus elastic workers so a job survives a lost machine.
4/ Observability: metrics, service-level objectives, and failure attribution across the other three.
Layers one and two keep hardware problems away from jobs. The third accepts that loss is unavoidable and makes it cheap. The fourth says whether the first three are working.
Health checks in two tiers
Device-level checks inspect each GPU. A node-level agent built on the open-source Kubernetes node-problem-detector runs on every GPU machine every 30 seconds and turns results into Kubernetes node conditions — the same mechanism used to mark a node NotReady. Controllers watch those conditions and decide whether the machine can accept work.
GPU device checks are built on NVIDIA DCGM for driver-reported error codes, uncorrectable memory errors, interconnect integrity, and thermal state. Networking adapter checks read host state for the count and link state of the Elastic Fabric Adapter interfaces carrying cross-machine traffic.
The properties that matter most:
- Failing closed. If the driver or diagnostic tool is unresponsive, the check times out and marks the machine unhealthy. Unreachable hardware reports a problem rather than passing, so a wedged driver leaves rotation instead of hiding behind a hung tool.
- A fallback path. The tool that reads driver error codes can itself break in certain failure modes. When it does, we read the kernel log
dmesgdirectly for those codes so no Xid or SXid errors are missed. - Our own fatal-error map. Vendor catalogs lag newer GPU generations and do not always match production. We keep our own mapping of which codes are fatal per GPU family — a code meaning reset-the-GPU in one generation can be benign in another, and one generation emits a burst of normally-fatal code on every healthy boot, which we treat as noise rather than drain fresh machines.
- No silent no-ops. A check fails when a required dependency is missing. An upgrade dropped a binary the network-adapter check relied on; the check quietly stopped running and machines with the wrong adapter count shipped for days before slow jobs were traced back. A missing dependency now fails the check.
From failed check to healthy capacity
A failed check changes the node condition, and controllers take over. Two levels are distinguished. A problem condition blocks new work from landing while jobs keep running, because evicting a job is disruptive and many problems clear on their own. Persistence past a tolerance window escalates to a failure condition, which evacuates the machine, drains its tasks (overriding normal disruption limits to move quickly), terminates it, and replaces it.
Karpenter node auto-repair is used on our compute stack, customized with a policy and controller that maps each node condition to the tolerated duration before action, so a fast-fatal error is actioned within minutes while a flaky signal gets a longer window. A cap limits how much of a cluster we disrupt at once, so a bad rollout of the checks cannot cascade into tearing down a whole cluster. We plan to upstream this feature to all Karpenter users.
Replacement is the slowest way to restore capacity, so the action is matched to the fault: killing the offending job clears a software wedge; resetting a single GPU clears many transient device faults; rebooting clears firmware state that drifted over long uptime; terminate-and-replace is reserved for hardware failing a test we trust. For that last case we record the GPU's immutable hardware identifier, which stays with the physical device across reboots and relaunches even when the instance changes, report it to the cloud provider, and keep our own list so a repeatedly failing device does not quietly rejoin under a new instance.
One subtlety sits between detection and scheduling: a freshly checked machine was handed to the scheduler while checks were still settling, so a job could land a moment before a slow check flagged it. Capacity is now held until checks have run and settled, closing that window.
Trusting a machine before it takes work
A just-booted machine is not trusted until it proves itself. Before capacity is released it passes system checks, confirms GPUs and their topology are present, and runs a round of GPU diagnostics. We then wait for the health checks to run twice, since longer checks such as adapter counting take more than one cycle to fire and a single pass would let work land before they report. Only then is the startup taint removed. Teardown runs deeper diagnostics and records GPU identifiers, feeding the same bad-hardware list so hardware that failed once does not silently cycle back.
Surviving the loss of a machine
Some machines are lost mid-job regardless, so that loss is made cheap: jobs checkpoint their state and resume from the last healthy checkpoint instead of restarting a multi-week run, workers are elastic and retryable, and the scheduler automatically retries platform-caused failures.
Attribution is what makes automatic retry safe. Retrying every failure would mask genuine job bugs and burn GPU hours looping on an error the platform cannot fix, so each failure is classified as platform or application before retry. One example: a class of timeouts in NCCL, the collective communication library GPUs use to exchange data, looked like a network fault and turned out to be data-loading stalls at high worker counts. The GPUs and network were fine and the fix was in the data pipeline; treated as hardware failures, those timeouts would have cycled healthy machines without fixing anything.
Measuring it: goodput over uptime
Three service-level objectives track GPU reliability, each split by cause so a miss points at the responsible layer:
- Availability — GPU capacity available for scheduling at any moment.
- Job-start success — whether a job starts promptly once its capacity is available.
- Job success — whether a job finishes without a platform-caused failure.
Causes are split into hardware, driver, platform, and user, so an availability dip from bad hardware reads differently from one caused by a scheduler change or a user misconfiguration. We optimize for goodput — the useful work the fleet completes — because a job held back by one slow GPU can look healthy on an uptime chart while finishing almost nothing. Specific targets for each objective are provided here for illustrative purposes.
The remediation loop, end to end
These pieces close a loop. Hardware is validated at allocation, before a job starts, with a short collective communication run and pairwise network tests, so any machine failing within the first minute fails before the job is admitted. During the job we watch passively for driver errors, memory errors, thermal events, and more, treating each job as a distributed system so we can identify a single slow rank dragging the rest rather than only whole-machine failures. Remediation is automatic and graded. The loop is the same whether a job runs through our batch scheduler Kueue or Ray.
Sharing GPUs without cross-tenant damage
Not every workload needs a whole GPU, and small inference jobs on dedicated devices waste capacity. GPUs are partitioned with MIG for memory-level isolation between tenants, so a fault or memory hog in one partition does not affect another. Partitions are reconfigured without restarting the machine, letting a pool shift between many small slices and a few large ones as demand changes. GPU and network adapter allocation is device-plugin-based today, moving to Kubernetes Dynamic Resource Allocation once supported in Karpenter — a GPU talking to an adapter or a CPU on the wrong PCIe root pays a bandwidth penalty.
Device-level allocation also changes how a single bad GPU is handled. With DRA support, the faulty device could be isolated while the others keep serving — most relevant for inference, where many small jobs share a machine. Expressive allocation also lets a job request the capabilities it needs, such as memory or tensor-core support, and the platform match a suitable GPU type, rather than pinning users to a model chosen in advance.
Results
Layered checks and automatic remediation changed two things users feel: more of the fleet is schedulable at any time because bad machines are caught and cycled quickly instead of sitting in the pool or waiting on an on-call engineer, and fewer jobs fail for platform reasons because most faults are caught before a job lands while the rest are remediated and retried. Graded remediation also returns capacity in minutes for faults needing only a job kill, GPU reset, or reboot, rather than the longer wait for a replacement.
Since these improvements landed:
- Consistently achieving >99.95% availability of our GPU nodes
- Job success rate has gone from ~98% to above 99.5%
Where this goes next
Work in progress extends active checks to more failure modes under load and widens burn-in so more failure classes are cleared before a user sees them. Remediation is becoming more granular so more error types are cleared in place rather than requiring a full node-replacement loop for certain Xid errors, and we want more periodic diagnostics between jobs.
Placement is moving toward topology awareness using Kubernetes Dynamic Resource Allocation once it lands in Karpenter — aligning on Compute/IMEX Domains, PCIe roots, and NUMA nodes.
For multi-GPU training performance, we plan a custom Linux scheduler built on sched_ext. First, prioritizing processes during collective coordination — when GPUs exchange results before the next compute round — to cut GPU idle time via a fleet-wide priority; the effect is small on a single machine but compounds across hundreds. Second, prioritizing data-loading processes to ease the bottleneck when data does not reach GPUs fast enough to sustain progress.
We already monitor throttling events, but a GPU need not report throttling for it to hurt a job: one running ten to fifteen degrees hotter than its peers, even within healthy limits, sets the pace for the gang. A large temperature gap can therefore be treated as a fault, stopping the job and moving it to cooler GPUs.
NCCL RAS (reliability, availability, serviceability) monitoring is also slated to expand to identify NCCL issues sooner, such as mismatched collectives or hangs, so work resumes faster.
Closing
Hardware fails, and at this scale it fails often enough that the platform must treat failure as normal. Detecting problems before and during jobs, remediating machines automatically, and letting long jobs resume after a loss lets teams run large-scale training and inference without babysitting the hardware. If you like these kinds of problems, we are hiring.
