Why Nodes Fail on Bare Metal
Cloudflare runs Kubernetes across five geographically diverse clusters, with hundreds of nodes in the largest one. These clusters are self-managed on bare-metal hardware, which gives us flexibility in software and Kubernetes integrations, but it also means no cloud provider is available to virtualize or manage the nodes for us. That distinction matters when nodes degrade, because the list of possible causes is long:
- Hardware failures
- Kernel-level software failures
- Kubernetes cluster-level software failures
- Degraded network communication
- Software updates that are required
- Resource exhaustion
A Network Interface Leak
One failure mode has been particularly persistent. It starts with a kernel log line:
unregister_netdevice: waiting for lo to become free. Usage count = 1The symptom is that the number of network interfaces owned by the Container Network Interface (CNI) plugin drifts out of proportion with the number of running pods:
$ ip link | grep cali | wc -l
1088This is unexpected because the count should never exceed the default maximum of 110 pods per node. The root cause is that Linux network interfaces owned by CNI are not cleaned up after a pod terminates. There is a long history of this issue, including a Docker GitHub issue that documents it. The problem tends to affect nodes with longer uptime; a reboot fixes it for about a month. On a significant number of nodes, however, it was occurring multiple times per day.
Each occurrence required a manual reboot cycle:
- Cordon the affected node to prevent new workloads from scheduling.
- Collect diagnostic information for later investigation.
- Drain current workloads.
- Reboot and wait for the node to return.
- Verify the node is healthy.
- Re-enable scheduling.
Solving the underlying issue would be best, but we needed a mitigation in the meantime to avoid the toil: automated node remediation.
Candidate Solutions
We limited automation to cases matching these criteria:
- Generic worker nodes
- Software issues confined to a single node
- Issues already researched and diagnosed
This excludes control-plane nodes, where remediation must account for etcd cluster health and redundancy for Kubernetes API components. We reviewed existing tooling with these constraints.
Node Problem Detector
Node problem detector runs as a daemon on each node and reports problems to the Kubernetes API. It supports pluggable problem daemons, so custom logic can be added. Problems are classified as permanent or temporary; permanent problems are stored as status conditions on node resources.
Draino and Cluster-Autoscaler
Draino drains nodes based on Kubernetes node conditions. It integrates with cluster-autoscaler, which adds or removes nodes through cloud provider plugins. That coupling made it a poor fit for bare metal.
Kured
Kured watches for the presence of a file on a node, then initiates drain, reboot and uncordon. It uses a Kubernetes API lock to ensure only one node is acted on at a time.
Cluster-API
The cluster-api project supports declarative cluster management with machine health checks. The checks use node conditions to detect unhealthy nodes and delegate replacement to the provider. Adoption would require a deeper investment than we wanted to make.
Proof of Concept
Most of these tools share a theme: they are pluggable components centered on Kubernetes node conditions. Node problem detector and Kured emerged as the most viable combination. We deployed node problem detector with a custom-plugin-monitor:
apiVersion: v1
kind: ConfigMap
metadata:
name: node-problem-detector-config
data:
check_calico_interfaces.sh: |
#!/bin/bash
set -euo pipefail
count=$(nsenter -n/proc/1/ns/net ip link | grep cali | wc -l)
if (( $count > 150 )); then
echo "Too many calico interfaces ($count)"
exit 1
else
exit 0
fi
cali-monitor.json: |
{
"plugin": "custom",
"pluginConfig": {
"invoke_interval": "30s",
"timeout": "5s",
"max_output_length": 80,
"concurrency": 3,
"enable_message_change_based_condition_update": false
},
"source": "calico-custom-plugin-monitor",
"metricsReporting": false,
"conditions": [
{
"type": "NPDCalicoUnhealthy",
"reason": "CalicoInterfaceCountOkay",
"message": "Normal amount of interfaces"
}
],
"rules": [
{
"type": "permanent",
"condition": "NPDCalicoUnhealthy",
"reason": "TooManyCalicoInterfaces",
"path": "/bin/bash",
"args": [
"/config/check_calico_interfaces.sh"
],
"timeout": "3s"
}
]
}Testing confirmed that the node's condition was updated when the problem occurred:
kubectl get node -o json worker1a | jq '.status.conditions[] | select(.type | test("^NPD"))'
{
"lastHeartbeatTime": "2020-03-20T17:05:17Z",
"lastTransitionTime": "2020-03-20T17:05:16Z",
"message": "Too many calico interfaces (154)",
"reason": "TooManyCalicoInterfaces",
"status": "True",
"type": "NPDCalicoUnhealthy"
}Kured handled the remediation side well, except that it watches a file rather than node conditions. We patched it to watch conditions instead and the proof of concept worked end to end.
Revisiting Detection
Node problem detector worked, but it was unwieldy: we were duplicating existing monitoring logic into shell scripts and configuration. Cloudflare's monitoring stack, described in a 2017 post, relies heavily on Prometheus and Alertmanager. For the network interface issue and similar problems, we already had metrics and alerts defined. Our alert looked like this:
- alert: CalicoTooManyInterfaces
expr: sum(node_network_info{device=~"cali.*"}) by (node) >= 200
for: 1h
labels:
priority: "5"
notify: chat-sre-core chat-k8sThe notify label drives Alertmanager routing. That raised a question: could alerts simply update node conditions instead of notifying humans?
Sciuro
Sciuro is our open-source replacement for node problem detector. It has one job: synchronize Kubernetes node conditions with currently firing alerts in Alertmanager. Node problems can be defined using existing exporters such as node exporter, cadvisor or mtail.
A key design difference: Sciuro does not run on the affected nodes. That makes out-of-band remediation possible. The high-level flow:
Prometheus scrapes node metrics and fires alerts to Alertmanager. Sciuro polls Alertmanager for alerts matching a specific receiver, matches them to Kubernetes node resources, and updates node conditions.
Here is the alert definition that triggers remediation:
- alert: CalicoTooManyInterfacesEarly
expr: sum(node_network_info{device=~"cali.*"}) by (node) >= 150
labels:
priority: "6"
notify: node-condition-k8sTwo changes from the previous alert: a new name with a more sensitive threshold, and a route to node-condition-k8s instead of chat. The intent is that automation tries to fix the node as quickly as possible; if the problem persists or worsens, humans still get notified at a higher severity.
Sciuro polls the Alertmanager API for alerts on that receiver. The equivalent using amtool:
$ amtool alert query -r node-condition-k8s
Alertname Starts At Summary
CalicoTooManyInterfacesEarly 2021-05-11 03:25:21 UTC Kubernetes node worker1a has too many Calico interfaces The alert labels, including node and instance, enable matching with the Kubernetes node:
$ amtool alert query -r node-condition-k8s -o json | jq '.[] | .labels'
{
"alertname": "CalicoTooManyInterfacesEarly",
"cluster": "a.k8s",
"instance": "worker1a",
"node": "worker1a",
"notify": "node-condition-k8s",
"priority": "6",
"prometheus": "k8s-a"
}Sciuro uses controller-runtime to track and update node resources. The updated condition is visible with kubectl:
$ kubectl get node worker1a -o json | jq '.status.conditions[] | select(.type | test("^AlertManager"))'
{
"lastHeartbeatTime": "2021-05-11T03:31:20Z",
"lastTransitionTime": "2021-05-11T03:26:53Z",
"message": "[P6] Kubernetes node worker1a has too many Calico interfaces",
"reason": "AlertIsFiring",
"status": "True",
"type": "AlertManager_CalicoTooManyInterfacesEarly"
}Sciuro prefixes condition types with AlertManager_ to avoid conflicts with kubelet-managed conditions such as DiskPressure. It also updates heartbeat and transition timestamps correctly. After the condition is set, existing tools such as our modified Kured handle the actual remediation.
Sciuro is open source on GitHub.
Managing Uptime Deliberately
Automatic remediation was initially meant for obvious problems, but we expanded its role to keep node uptime low deliberately. Lower uptime reduces configuration drift, keeps the node initialization process tested, and encourages high-availability deployment practices. Services deployed with proper redundancy handle a single node leaving the cluster cleanly; those relying on singleton pods expose their fragility.
This alert drives the uptime management:
- alert: WorkerUptimeTooHigh
expr: |
(
(
(
max by(node) (kube_node_role{role="worker"})
- on(node) group_left()
(max by(node) (kube_node_role{role!="worker"}))
or on(node)
max by(node) (kube_node_role{role="worker"})
) == 1
)
* on(node) group_left()
(
(time() - node_boot_time_seconds) > (60 * 60 * 24 * 7)
)
)
labels:
priority: "9"
notify: node-condition-k8sThe alert uses kube_node_roles to target generic worker nodes and checks node_boot_time_seconds from node exporter. The notify label routes to node conditions. Priority is set to "9", lower than other alerts. The message field is prefixed with the priority in brackets so the remediation process can decide which node to handle first — important because Kured's lock allows only one node to be remediated at a time.
Field Results and Next Steps
Over the past 30 days, this automatic remediation process has handled 571 nodes without human intervention. The measurable benefits have been twofold: a significant reduction in manual toil for the operations team, and faster repair times across the board, since automated remediation is always on call and can respond more quickly than a human paging rotation.
What’s Coming for Sciuro
We’re open sourcing Sciuro on GitHub, and we welcome issues, suggestions, and pull requests. Our near-term roadmap for Sciuro itself focuses on latency reduction. The current design relies on polling, which introduces some delay; we’re considering moving to a push-based model from Alertmanager, though it hasn’t become a necessity yet.
Redesigning the Remediation Layer
The broader architecture around node remediation is also slated for an overhaul. We’re currently running on a fork of kured, and we have clear requirements for a future replacement component:
- Integrate with out-of-band management interfaces so we can power nodes down and back up even when the operating system is unresponsive.
- Shift from a decentralized model to a centralized one, which would allow us to orchestrate more sophisticated logic like acting on entire failure domains in parallel.
- Add explicit support for specialized node types, including control plane masters and storage nodes.
If this kind of work sounds interesting, we're actively looking for more Kubernetes engineers to join our team.
1Exhaustion can be applied to hardware resources, kernel resources, or logical resources like the amount of logging being produced.
2Almost all Kubernetes objects carry spec and status fields. The status field describes the object’s current state. For nodes, the kubelet typically populates a conditions field under status to report things like whether the node is ready to schedule pods.
3The alert format used here is documented in the Prometheus Alerting Rules reference.
