Java 21 Virtual Threads - Dude, Where’s My Lock?

Virtual threads in production: a deadlock post-mortem

Netflix’s JVM Ecosystem team routinely evaluates new language features as we move across Java versions. Virtual threads, introduced in Java 21, were high on our list: they promise high-throughput concurrency without the complexity of reactive programming. Our optimism was tempered by an intermittent production issue that surfaced shortly after we began enabling virtual threads in select services.

A familiar failure signature

Several teams reported timeouts and hung instances. The affected services all ran Java 21 with SpringBoot 3 and embedded Tomcat. The JVM stayed alive, but the instance stopped serving traffic. A telltale graph showed sockets accumulating in the closeWait state, indicating that remote peers had closed connections but the local application never closed its end.

We suspected a hang, but standard jstack thread dumps showed an idle JVM. The reason: virtual thread stacks are invisible to jstack. We switched to jcmd Thread.dump_to_file and also collected a heap dump for deeper inspection.

Thousands of blank threads

The jcmd dumps revealed thousands of “blank” virtual threads — thread objects created but never started, with no stack traces. The count roughly matched the number of closeWait sockets. To interpret this, recall how virtual threads work: they are tasks scheduled onto a fork-join pool of OS threads. When a virtual thread blocks, it unmounts from its carrier thread, freeing that carrier for other virtual threads.

In our setup, Tomcat runs in a blocking model. Enabling virtual threads makes Tomcat create a new virtual thread per request via a VirtualThreadExecutor. The blank threads were requests that had been accepted but never executed — Tomcat kept creating threads, but no carrier thread was available to run them.

Pinned and stuck

The carrier threads were not idle; they were pinned. A virtual thread pins to its carrier when it blocks inside a synchronized block or method. Our thread dumps showed four virtual threads pinned while waiting to acquire a reentrant lock in brave.RealSpan.finish(). Since the affected instance had 4 vCPUs, the fork-join pool had exactly 4 OS threads — all occupied by pinned virtual threads. No other virtual thread could run, so Tomcat stopped processing requests while the socket backlog kept growing.

A fifth virtual thread was waiting on the same lock but was not pinned. A sixth thread, a normal platform thread, was also blocked. The platform thread’s stack showed it had entered awaitNanos() while owning the lock, but after the wait completed it could not reacquire it.

The missing lock owner

With five virtual threads and one platform thread all waiting on the same ReentrantLock, the obvious next question was: who holds the lock? Standard thread dumps annotate lock ownership with - locked <0x…> or Locked ownable synchronizers, but neither appeared. Java 21’s jcmd dumps do not include this information; the omission is slated for correction in future releases.

With thread dumps exhausted, we turned to the heap dump to introspect the lock’s internal state directly. The lock was a ReentrantLock used in a CountBoundedQueue — a bounded queue that coordinates between a producer and a consumer. Reconstructing the lock’s state from the heap revealed the full picture.

The deadlock sequence was as follows. One thread owned the lock and was inside awaitNanos(), waiting for a slot to become available in the bounded queue. Before that wait completed, another thread attempted to acquire the same lock inside a synchronized block in the tracing library. That second thread pinned its carrier thread while waiting. As more requests arrived, additional virtual threads followed the same path and pinned the remaining carriers. Once all 4 carriers were pinned, no virtual thread — including the one that owned the lock and needed to release it — could be scheduled. The locking thread never resumed, the condition never signaled, and the system deadlocked.

Notably, this was not a classic deadlock where each thread holds a resource and waits for another. It was a scheduling deadlock unique to virtual threads: the lock owner was never scheduled because all carriers were held by waiting threads that could not proceed until the lock was released. The platform thread’s inability to reacquire the lock after awaitNanos() was a symptom of the same root cause — the lock owner, a virtual thread, was starved of a carrier.

Heap-dump analysis of the lock's AbstractQueuedSynchronizer state allowed us to identify the exact thread that held the lock and confirm the diagnosis. With the threads that were blocked, the code path that created the pinning condition, and the missing lock owner all known, the fix was clear: the tracing code needed to avoid blocking operations inside synchronized blocks. Our workaround involved restructuring the span-finishing path to release the monitor before any potentially blocking operations, and we are tracking the upstream library change to eliminate the pinning entirely.

Reading the heap dump

The lock object itself was easy to locate in the heap dump using Eclipse MAT: we walked the stack of the AsyncReporter non-virtual thread and identified the object it was blocked on. Interpreting the lock's current state took more effort. The relevant code lives in AbstractQueuedSynchronizer.java; we reverse-engineered just enough of it to match against the dump.

Three observations stand out. First, exclusiveOwnerThread is null (2), so no thread currently owns the lock. Second, the head of the wait queue is an "empty" ExclusiveNode (3) — its waiter is null and status is cleared — followed by a second ExclusiveNode whose waiter points at virtual thread #119516 (4). Third, state = 0 (1) confirms the lock is free.

The only code path that nulls out exclusiveOwnerThread is ReentrantLock.Sync.tryRelease() (source link), which also sets state = 0 — matching what we see. Tracing the caller chain from there, the release path proceeds to signal the next waiter after tryRelease() succeeds. At that moment the releasing thread still occupies the head node even though ownership is effectively gone; the following node references the thread about to acquire the lock.

The acquire path is, roughly, an infinite loop in AbstractQueuedSynchronizer.acquire(): try to grab the lock, park on failure, retry when unparked.

while(true) {
   if (tryAcquire()) {
      return; // lock acquired
   }
   park();
}

When the releasing thread unpark the next waiter, that unparked thread loops around and attempts acquisition again. Indeed, all of our contending threads were parked on line 754. A thread that then succeeds in acquiring the lock resets the head of the list and clears its waiter reference via this block.

Put together: the head node references the lock owner; releasing notifies the next node; acquiring resets the head. The heap dump therefore captures the brief interval after one thread has released but before the next has acquired — an in-between state that should be transient. Our JVM is stuck there. Thread #119516 was notified and is next in line, per the ExclusiveNode state. Yet the thread dump shows #119516 still parked, along with every other contender. The two dumps contradict each other — unless the unparked thread cannot actually run.

Nowhere to schedule

With that hypothesis, we re-read the thread dump more carefully. Six threads are waiting on the lock. Four are virtual threads each pinned to an OS thread; they will not release their carriers until they acquire the lock and exit the synchronized block. Thread #107 "AsyncReporter " is a regular platform thread, so nothing blocks it once it gets the lock. The last waiter, #119516, is a virtual thread that is not pinned — but the fork-join pool has no free carriers left. The four pinned VTs occupy all OS threads, and they cannot proceed without the lock. Thread #119516 receives its unpark signal but cannot be scheduled onto a carrier to act on it. This is a classic deadlock variant: instead of two locks, we have one lock and a semaphore of four permits represented by the fork-join pool.

Once the failure mode was clear, a minimal reproducer was straightforward to write.

Closing thoughts

Virtual threads in Java 21 deliver on their core promise of cheaper concurrency, but this case shows a sharp edge in the interaction between them and classic locking primitives when the carrier pool is exhausted. The Java 23 release train brings improvements we hope will smooth out this integration. Investigations like this are one slice of what performance engineering looks like day-to-day; we share the process in the hope it aids similar debugging elsewhere.