What a hung task warning really means

When the Linux kernel logs a hung task warning, it means some process has remained in the TASK_UNINTERRUPTIBLE state — the D state — for longer than the configured timeout. That state exists to protect process memory: a task in it cannot be interrupted by signals, and only a specific wake-up event or a reboot can clear it. The design makes sense for cases where data consistency matters, such as an ongoing disk write that must complete before the process can safely terminate.

The state is intentionally unresponsive, which is what makes it both useful and dangerous. If the wake-up event never arrives, or arrives late, the process stays hung. Complications arise when such a process holds a lock that others need, or when many processes pile up in the D state at once — a sign that some resource is overwhelmed or malfunctioning. The kernel's answer to the rough edges of TASK_UNINTERRUPTIBLE is TASK_KILLABLE, which still preserves the process but allows termination by a fatal signal.

How the kernel detects hung tasks

The khungtaskd kernel thread periodically scans all processes and flags any that have been in the D state beyond the threshold. The relevant tunables are:

$ sudo sysctl -a --pattern hung
kernel.hung_task_all_cpu_backtrace = 0
kernel.hung_task_check_count = 4194304
kernel.hung_task_check_interval_secs = 0
kernel.hung_task_panic = 0
kernel.hung_task_timeout_secs = 10
kernel.hung_task_warnings = 200

Cloudflare changed kernel.hung_task_timeout_secs from its 120-second default to 10 seconds, and caps kernel.hung_task_warnings at 200 messages, resetting the counter every 15 minutes. Setting the warnings value to -1 makes it unlimited.

Case study: XFS kworker pressure

Hung task warnings usually name the affected process, but sometimes they show a kworker thread instead. Kernel workqueues aggregate deferred work from many different tasks, which obscures which application is actually delayed. The accompanying Workqueue line helps: with a workload named xfs-sync, the hint points toward xfs_log_worker and the XFS filesystem subsystem.

BLOG-2660 hero image

In this case, the XFS subsystem was under heavy pressure. The alerts led to discovering that configuration changes had dropped the no_read_workqueue/no_write_workqueue flags previously set to speed up Linux disk encryption. No fatal condition existed, but the warning correctly flagged a filesystem that had slowed down.

Case study: coredump delays

When a process terminates abnormally and coredump handling is enabled, the kernel snapshots the process memory before exiting and hands it to a handler such as systemd-coredump. During this operation, the kernel moves the process to the D state to protect its memory. Larger memory footprints mean longer coredump times — and a higher chance of tripping the hung task threshold.

A stack trace pointing to coredump_task_exit confirmed the behavior in practice. Testing with a small Go program that read a 10 GB file into memory and then crashed produced exactly this warning:

$ sudo dmesg -T | tail -n 31
INFO: task test:8734 blocked for more than 22 seconds.
      Not tainted 6.6.72-cloudflare-2025.1.7 #1
      Blocked by coredump.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:test            state:D stack:0     pid:8734  ppid:8406   task_flags:0x400448 flags:0x00004000

A recent upstream addition makes such cases easier to identify: a log line reading Blocked by coredump., backed by the PF_POSTCOREDUMP task flag, clarifies that the process isn't at fault — the coredump mechanism is simply taking time.

The warning can identify a victim, not the offender. The application in the log may be perfectly healthy; the real problem may be in kernel machinery acting on its behalf.

Case study: a held rtnl_mutex

Sometimes hung task warnings arrive in bulk across many unrelated processes. When dozens of tasks stall for minutes with no improvement, it's worth examining their stack traces for common patterns. In one incident, every trace converged on three functions, all waiting for rtnl_lock — a global mutex protecting network configuration. Some task had acquired it and was not letting go.

The hung task reports show the victims stuck waiting, but not the actual lock holder. A bpftrace script can expose the culprit by inspecting the mutex's owner field — encoded as atomic_long_t with state bits in the three lowest bits, so it needs masking before dereferencing:

#!/usr/bin/env bpftrace
interval:s:10 {
  $rtnl_mutex = (struct mutex *) kaddr("rtnl_mutex");
  $owner = (struct task_struct *) ($rtnl_mutex->owner.counter & ~0x07);
  if ($owner != 0) {
    printf("rtnl_mutex->owner = %u %s\n", $owner->pid, $owner->comm);
  }
}

The script periodically reads the rtnl_mutex owner via kaddr() and printed the owning task's name:

rtnl_mutex->owner = 3895365 calico-node

That output pointed at calico-node. Reading /proc/PID/stack showed the process was inside wg_set_device(), with peer_remove_after_dead() blocked on a NAPI disable — a Wireguard configuration change held the RTNL lock. Further investigation was left to the upstream kernel community, but the hung task reports had narrowed the problem space considerably.

Debugging guidance

  • Start with the stack trace, even when the messages look disconnected. Patterns may emerge that link many warnings to a single root cause, as in the RTNL case.
  • Treat each warning as a possible misdirection. The logged process is not always the offender — coredump handling and lock holders can both make innocent tasks look guilty.
  • When the kernel withholds CPU time by placing a task in the D state, there's usually a legitimate reason, but the underlying cause may be application code, kernel configuration, or a subsystem failure.