Kafka consumers can look alive while doing nothing — offset-aware probes fix that

Cloudflare relies on Kafka for time-sensitive workloads such as alerts and transactional emails, and keeps those workloads healthy through Kubernetes-managed microservices with health checks. But the conventional way of probing Kafka consumers — verifying the connection to the broker — misses the failure mode that matters most: a consumer that is connected yet no longer making forward progress.

How Kafka consumption works at scale

Kafka topics hold an ordered log of events. Each log is split across partitions, and those partitions are spread across brokers. Consumer groups coordinate so that each partition is assigned to one consumer; commits record the offset of the last processed message, and lag is the gap between the latest message and the consumer's position. At Cloudflare's message rates, that gap can widen quickly, so monitoring it continuously is essential.

Consumers run as Kubernetes pods, which makes liveness probes the natural place to enforce health. Basic probes check broker connectivity, but at Cloudflare's scale a typical topic can have hundreds of partitions with a replica count that doesn't match. A connection check can pass while a consumer is stuck waiting on messages for a partition it is no longer assigned — no processing, no commits, no restart.

Why idle consumers escape detection

A consumer reporting ready and reading from a topic is not the same as a consumer processing messages. Manually restarting a stuck pod triggers a rebalance, where the group leader redistributes partitions. In Cloudflare's library implementation, a consumer that fails to acknowledge the stop signal may keep waiting on a partition that was reassigned — a deadlock that usually needs a manual restart.

The startup probe pattern that looks only at broker responses treats that deadlocked consumer as healthy. The fix is to make the probe ask a different question: is the consumer advancing its committed offsets?

Offset-based liveness logic

Inspired by PagerDuty's Kafka health-check design, Cloudflare's probe compares the topic's current and committed offsets. The current offset is the last message published; the committed offset is the last message processed by the consumer group. Evaluating both gives the probe four rules:

  • Fail if the current offset cannot be read.
  • Fail if the committed offset cannot be read.
  • Pass if the committed offset equals the current offset — the consumer is caught up.
  • Fail if the committed offset hasn't changed since the previous probe run, indicating no forward progress.

To detect movement between runs, the service keeps an in-memory map of the last observed committed offset per partition. Each replica tracks only the partitions it currently consumes.

Rebalances exposed a flaw in the first version

The initial rollout triggered cascading failures. When a rebalance moved a partition away from a replica, that replica's in-memory data still held the partition's old committed offset. On the next probe run the service saw the value hadn't changed — but no replica was updating it anymore — and declared itself unhealthy. Restarting that pod triggered another rebalance, which passed the problem to other replicas.

The solution relies on the Shopify Sarama library's rebalance notifications. By listening on the session context channel, the probe rebuilds the in-memory offset map when a rebalance signal arrives, keeping only the partitions the replica currently owns. Validation involved scaling a single-replica consumer up to match partition count and back down to one. The probe correctly handled both partitions being added and partitions being taken away without false health failures.

A health check should define the service's own failure state

Kubernetes probes are inexpensive and powerful when they assert the narrow contract of the workload. Broad checks that only confirm external dependencies are connected will pass while the workload stands still, producing a false sense of security. The better approach is to reason about what unhealthy means for each specific service and build the probe around that — for a Kafka consumer, it’s progress in the offset log, not just an open connection.