A Fine-Grained Window Into Network Traffic
Millisampler is Meta’s lightweight network traffic characterization tool for continual monitoring. Designed to observe, characterize, and debug network performance at high-granularity timescales, it operates on the server fleet, collecting short, periodic snapshots at configurable intervals of 100μs, 1ms, and 10ms. The tool stores this data on local disk, where it remains available for several days for on-demand analysis.
The tool gathers time series data for ingress and egress traffic volumes, the number of active flows, incoming ECN marks, and ingress/egress retransmissions. It can also distinguish between in-region traffic and cross-region traffic (the latter having longer RTTs). While the collection uses only aggregated flow-level header information—containing no personally identifiable information (PII)—it has proven highly useful in practice. When combined with existing coarser-grained data, Millisampler's fine timescale view helps reveal when network components like switch buffers or host NICs are unable to handle traffic patterns.
Kernel Collection, Userspace Control
Millisampler is split between userspace and kernel components. Userspace code schedules runs, stores data, and serves it; an eBPF-based tc filter in the kernel performs the fine-timescale data collection. The user code attaches the tc filter and enables data gathering. The filter is positioned among the first programmable steps on packet receipt and near the last step on transmission. On the ingress path, the eBPF code executes on the CPU core processing the soft IRQ (bottom half) as the packet heads to its owning socket.
Because packet processing spans many CPU cores, Millisampler uses per-CPU variables to avoid locks, accepting increased memory requirements to eliminate contention risk. To minimize overhead, sampling happens periodically and for short durations. Userspace configures two key parameters: the sampling interval and the sample count. In production, runs are scheduled with three intervals—10ms, 1ms, and 100μs—and a fixed 2,000 samples per interval. This yields observation periods from 200ms (at the 100μs rate) to 20s (at the 10ms rate). These windows let operators observe events from sub-RTT to cross-region RTT timescales, while fixing the memory footprint at 2,000 64-bit counters per CPU core for each measured value.
The tool computes ingress and egress total bytes, plus ingress ECN-marked bytes, from packet lengths and CE bits. It also sounds TTLd-marked retransmits. For connection counting, Millisampler uses a 128-bit sketch to estimate the number of active incoming and outgoing connections. This approach approximates the connection count with precision up to a dozen connections, saturating at roughly 500 connections per sampling interval. Although additional precision is possible, the qualitative distinction between a few connections and dozens or hundreds has proven most valuable — identifying patterns like heavy incast (many connections) versus fewer connections but bulkier traffic.
Diagnosing Faults and Characterizing Services
Millisampler has become a key tool for troubleshooting and performance analysis. Two contrasting network performance faults at Meta illustrate its value. The first involved synchronized traffic bursts at fine timescales; observing this pattern motivated the development and deployment of Millisampler to catch recurrences quickly. The second—a NIC driver bug that halted packet delivery for milliseconds at a time—was root-caused with the help of an early prototype, validating the tool's role in complex investigations. In both cases, Millisampler worked alongside a larger ecosystem of data collection tools tracking a broad range of metrics across hosts and the network.
Beyond incident response, Millisampler data supports service traffic characterization. It has helped quantify the intensity of incast bursts across services, guiding transport performance tuning, and has made visible the complex interactions between short-RTT and long-RTT flows—specifically, how bursts from one affect fairness for the other. A related tool, Syncmillisampler, runs Millisampler synchronously across all hosts in a rack, using the synchronized data to identify buffer contention in top-of-rack ASICs.
