Investigation of a Workbench UI Latency Issue - Netflix TechBlog

When a JupyterLab UI Stalls: A Debugging Trip to the Linux Kernel

Netflix’s Workbench, a remote development environment built on the Titus container platform, is a common home for JupyterLab notebooks used in big data and machine learning work. Recently, users reported that the JupyterLab interface became sluggish and unresponsive while running particular notebooks. Restarting the ipykernel process provided only temporary relief. The challenge was to turn a vague, subjective complaint into a measurable problem, then trace it from the browser all the way down to the operating system.

Quantifying UI Responsiveness

To measure the UI slowness objectively, we opened a terminal in JupyterLab and held down a key (e.g., “j”) for 15 seconds while the problematic notebook ran. The terminal’s stdin is sent to the backend over a WebSocket, and stdout is returned and rendered in the UI. We exported a .har file capturing all browser-server communications and analyzed it in a notebook.

This approach revealed latencies ranging from 1 to 10 seconds, with an average of 7.4 seconds.

Examining the Suspect: pystan

A natural first suspect was the code inside the “bad” notebooks. One library stood out: pystan, a Python binding for the native C++ Stan library. pystan relies on asyncio, but since a notebook process already has a running event loop, the library’s authors recommend using nest_asyncio to inject its operations into the existing loop. This unmaintained library felt hacky, and we suspected it might be blocking WebSocket message handling.

However, a closer look revealed why this theory was likely wrong. The notebook executes in a child process—the ipykernel process—of the jupyter-lab server. Injecting events into the ipykernel event loop should not impact the jupyter-lab main event loop that handles UI WebSocket traffic.

Packet captures on the ZeroMQ socket between the kernel and the server also showed no heavy traffic that could cause blocking. The strongest evidence, though, came later: we reproduced the issue in a notebook that did not use pystan at all.

Other Dead Ends: CPU and Network

Next, we considered the possibility that the Workbench container was suffering from CPU starvation due to “noisy neighbors” on the host. Titus uses CPU oversubscription, so containers can compete for physical CPU resources. But using top, we saw that the Workbench used only 4 of its 64 allocated CPUs during the problematic notebook run. Elapsed time was not the bottleneck.

We then examined the network path between the browser and the server. Packet captures taken while pressing the key showed a 5-second pause in traffic from server port 8888—but port 22 (SSH) traffic was unaffected. That pointed directly at the JupyterLab process itself, not the network.

A Surprising Minimal Reproduction

By stripping the “bad” notebook down, we found a minimal case that required no third-party dependencies:

import time
import os
from multiprocessing import Process

N = os.cpu_count()

def launch_worker(worker_id):
  time.sleep(60)

if __name__ == '__main__':
  with open('/root/2GB_file', 'r') as file:
    data = file.read()
    processes = []
    for i in range(N):
      p = Process(target=launch_worker, args=(i,))
      processes.append(p)
      p.start()
 
    for p in processes:
      p.join()

This code does two simple things:

1. Reads a 2GB file into memory (negligible for the 480GB available).
2. Starts N processes—where N equals the number of CPUs—that just sleep.

Neither CPU-bound nor memory-bound, this seemingly trivial code could still stall the JupyterLab UI for up to 10 seconds. The odd behavior raised several questions:

• Both steps were required; skipping the file read made the issue disappear. Why did using a small fraction of memory matter?
• During the delay, the jupyter-lab process’s CPU utilization spiked to 100%, even though it wasn’t running the notebook.
• How could activity in a child process cause CPU contention in its parent?

One more puzzle: the code started more processes than there were CPUs. Workbench reported 64 CPUs, but os.cpu_count() returned 96. This discrepancy stems from a missing “CPU namespace” in the Linux kernel. The containerized environment leaks the host’s physical CPU count (96) when using certain system calls, while commands like nproc correctly report the allocated virtual CPUs (64). Python’s os.cpu_count() follows the former path, an issue Python 3.13 addresses with a new accurate API.

Profiling the Parent Process

The breakthrough came from profiling the jupyter-lab process with py-spy—not the ipykernel child that ran the reproduction code.

The profile showed 89% of CPU time spent in a function called __parse_smaps_rollup, while the terminal handler used only 0.47%. This function ran inside event loop A and was invoked by the jupyter_resource_usage extension. Disabling this extension eliminated the UI slowness completely.

The root cause became clear: the resource usage extension, which monitors memory consumption, was the culprit. The odd combination of reading a file and spawning processes triggered a code path in this extension that caused the parent jupyter-lab process to parse smaps_rollup entries with pathological inefficiency. The deeper investigation into why that parsing is so expensive, and why this workload triggers it, reveals an unexpected interaction between a simple notebook, a UI extension, and the Linux kernel’s memory accounting.

Why More CPUs Made the Workbench Slower

Once a workbench user launched a Notebook that spawned dozens of child processes and read a 2GB file, the entire UI froze. Thread dumps from the hung Jupyter process pointed to a function from the jupyter-resource-usage extension, which the UI calls periodically via the /metrics/v1 API endpoint.

The extension's get method (at jupyter_resource_usage/api.py:42) recursively enumerates every child process of the jupyter-lab process, including the ipykernel process and all processes created by the Notebook. The cost is linear to the total number of descendant processes. With a machine allocation of 64 CPUs, normal operation means about 66 processes. But starting 96 sleep processes from the Notebook — one per visible CPU, even though only 64 were allocated — brought the total closer to 98, and with it a proportionally slower UI.

The more CPUs you have, the more processes the Notebook might spawn, and the slower the monitoring call becomes.

The Second Half: Reading a Large File

Starting many child processes alone didn't reproduce the issue. The slowness required the child process to read a 2GB file into memory. The key is in __parse_smaps_rollup from the psutil library, which reads /proc/<pid>/smaps_rollup for every process.

The number of lines in this file is constant, regardless of memory usage, and it's a kernel interface, not a regular disk file. But the file's read handler in the kernel — introduced in 2017 to speed up aggregate memory statistics — walks the process's virtual memory areas in a do-while loop. The duration of that loop is linear to the number of virtual memory areas. Reading a 2GB file creates enough additional memory maps to dramatically slow down each smaps_rollup read.

Quantifying the Penalty

To measure the impact directly, we read the current process's smaps_rollup, then loaded a 2GB file, then read the file again, timing each read with strace:

• Before loading the file: read syscall took 0.000259 seconds.
• After loading the file: read syscall took 0.027698 seconds — about 100x slower.

Both calls returned the same 670 bytes from smaps_rollup. With 98 child processes, the aggregate cost of reading this file for each process was roughly 2.7 seconds per refresh cycle — enough to stall the UI.

The Fix

The jupyter-resource-usage extension is what powers the CPU and memory indicator in the Notebook's bottom bar. For this user, that indicator wasn't essential. Disabling the extension restored full UI responsiveness and resolved the issue.

What We Learned

The breakdown occurred across two independent dimensions: the extension's process tree traversal scales linearly with the CPU count, while the kernel's smaps_rollup handling scales linearly with virtual memory size. The convergence of these two linear costs — neither of which is usually a bottleneck on its own — can grind a workbench UI to a halt.

The irony cuts both ways: the very tool meant to monitor CPU usage triggered CPU contention, and having more CPUs made performance worse, not better.