Transparent memory offloading: more memory at a fraction of the cost and power

The DRAM squeeze and the case for offloading

Meta's data center fleet is growing faster than DRAM capacity can keep pace. Memory-hungry workloads—machine learning chief among them—are colliding with a slowdown in DRAM scaling and volatile pricing, making DRAM untenable as the sole source of memory at hyperscale. Meanwhile, NVMe-connected SSDs and other non-DRAM technologies deliver far more capacity per dollar and per watt. The technical question is no longer whether to offload colder memory to cheaper media, but how to do it robustly across heterogeneous fleets without hurting application performance.

The answer Meta has built and run in production since 2021 is Transparent Memory Offloading (TMO), a kernel-driven solution that has so far saved 20 to 32 percent of total memory per server across millions of machines. TMO's core insight is that memory pressure can be measured directly as lost work, and that offloading can be tuned in real time against that signal—no application knowledge required. The OS components have been upstreamed to the Linux kernel.

A hierarchy worth exploiting

Memory tiering is conceptually simple: migrate less-frequently-accessed pages to slower storage. The migration can be driven by the application, a userspace library, the kernel, or the hypervisor. TMO chooses kernel-driven swapping because it applies transparently to unmodified applications. The memory-storage hierarchy that makes this attractive spans DRAM at the top, compressed memory and NVM in the middle, and NVMe SSDs below.

The economics are stark. DRAM's share of server cost is climbing and projected to reach 33 percent of infrastructure spend; power follows a similar trajectory toward 38 percent. Compressed DRAM, assuming a 3x compression ratio representative of Meta's production workloads, offers a cheaper middle tier. But NVMe SSDs are even more compelling: under 3 percent of server cost at the system level—about 3x cheaper than compressed memory in the current server generation—and under 1 percent of server cost iso-capacity to DRAM, roughly 10x cheaper per byte than compressed memory.

The performance gap is real, but access patterns create the opportunity. Across Meta's fleet, about 35 percent of application memory consists of pages not touched in the past five minutes—safe to offload without visible impact. That average hides wide variance, from 19 percent to 62 percent depending on the workload, which underscores the need for an adaptive mechanism rather than a fixed policy.

Offloading must also handle both kinds of application memory. Anonymous memory—heap and stack pages allocated directly by the application—behaves differently from file-backed memory held in the kernel's page cache. Production workloads span the spectrum from almost purely anonymous to page-cache-dominated, so TMO has to treat both fairly.

Measuring pressure the right way

Traditional memory-health metrics like page fault rates are misleading at scale. Fault rates spike naturally on cold caches or working-set transitions, and the same fault rate means very different things depending on whether the backing store is a spinning disk or a fast SSD. What TMO needed was a metric that captured the actual cost of a memory shortage to the workload.

That metric is Pressure Stall Information (PSI), a Linux kernel component that tracks threads stalled specifically because of memory—for example, a thread faulting on a recently reclaimed page or entering reclaim to satisfy an allocation. PSI aggregates thread states into two indicators: some, where one or more threads are stalled, and full, where all non-idle threads are simultaneously stalled and no work can progress. The time spent in these states is reported as a percentage of wall clock.

If a container's full metric reads 1 percent over a 10-second window, the workload lost 100ms of fully productive time to memory shortage. The underlying event rate is irrelevant: 10 faults on a hard drive or 10,000 on an SSD produce the same PSI reading if the lost time is equal. PSI gives TMO a direct, comparable signal of memory-induced slowdown across heterogeneous hardware.

Senpai: a pressure-guided control loop

At the heart of TMO is Senpai, a userspace agent running a control loop over PSI metrics. If a container's pressure is below a calibrated threshold—one low enough that paging overhead doesn't functionally degrade the workload—Senpai increases the reclaim rate. If pressure rises above target, it eases off. The reclaim amount is computed as a function of current memory usage, a reclaim ratio, and the deviation from the pressure target:

reclaim = current_mem * reclaim_ratio * max(0,1 – psi_some/psi_threshold)

Senpai acts every six seconds, a cadence that lets reclaim activity translate into measurable pressure through subsequent refaults.

The original implementation drove reclaim by lowering the cgroup2 memory limit, but that approach had critical failure modes. If Senpai crashed, it left behind a restrictive limit that could cause extreme pressure or OOM kills. Even without crashes, Senpai often couldn't raise the limit quickly enough to accommodate rapidly growing workloads, producing pressure spikes beyond tolerance. The fix was a new stateless kernel interface: the memory.reclaim cgroup control file. It lets Senpai request reclaim of an exact number of pages without imposing any persistent limit, eliminating the risk of blocking expanding workloads.

A swap algorithm for the flash era

Even with PSI and Senpai in place, Linux presented a stubborn obstacle: it refused to swap anonymous memory at the subliminal pressure levels TMO targets. The kernel would happily evict file cache under pressure, but configured swap space sat idle even when cold heap pages existed and the file cache was thrashing.

The behavior is a relic of the spinning-disk era. Linux evolved when storage consisted of hard drives with seek times that made the semirandom I/O patterns of swapping brutally slow. With memory sizes growing and disk I/O rates stagnant, paging significant fractions of a workload was futile. Swap became an emergency measure reserved for near-OOM conditions.

Contemporary flash changes that calculus. Even commodity SSDs handle hundreds of thousands of IOP/s—orders of magnitude beyond high-end hard drives, which struggle to reach a hundred. Paging a few gigabytes across such a device is routine.

TMO's new swap algorithm exploits flash capability without regressing legacy rotational-media setups. It tracks the rate of file cache refaults and engages swap in direct proportion: for every file page that repeatedly must be read back from the filesystem, the kernel swaps out one anonymous page. This frees room for the thrashing file page. If swap-ins occur, reclaim pressure shifts back onto the file cache.

The feedback loop settles at an equilibrium that evicts the coldest memory overall across both pools, serving the workload with minimal aggregate paging I/O. Because it only ever trades one form of paging for another, it cannot perform worse than the prior algorithm. In practice, the kernel begins swapping at the first signs of file cache distress, putting otherwise idle swap space to work at TMO's low pressure levels.

Making the memory tax disappear

TMO's production deployment has now surpassed one year, yielding substantial fleet-wide reductions in memory consumption. Meta breaks these gains into two categories: savings from the primary applications themselves, and savings from the "memory tax"—the overhead from data center and application management services resident on each host.

For application-level savings, results vary by offload back end due to data compressibility. With a compressed-memory back end, TMO reduces resident memory by 7 to 12 percent across five representative applications. However, certain workloads, such as quantized byte-encoded machine learning models used in Ads prediction, exhibit poor compressibility—typically a 1.3–1.4x compression ratio—making SSD offloading more effective. For these applications, SSD-backed offloading achieves 10 to 19 percent savings. Across both back ends, TMO consistently delivers 7 to 19 percent total memory savings with no discernible impact on application performance.

The memory tax reductions are equally meaningful. By offloading overhead from fleet-wide management services unrelated to the primary workload, TMO achieves an average of 9 percent memory savings per server from the data center tax and another 4 percent from the application tax. Combined, the 13 percent average tax reduction is additive to workload savings and represents considerable memory at fleet scale.

From manual tuning to a memory hierarchy

Today, selecting the offload back end—compressed memory versus SSD-backed swap—is a manual decision based on an application's memory compressibility and its tolerance for slower memory access. Automating this choice with tooling is possible, but Meta sees a more fundamental solution: the kernel managing a hierarchy of back ends. In this model, the reclaim algorithm would automatically route warmer pages to zswap, colder or less compressible pages to SSD, and eventually fold NVM and CXL devices into the hierarchy. The kernel's reclaim logic would then dynamically balance memory across these pools—an architecture that is currently in active development.

Looking ahead, bus technologies like CXL offer memory-like access semantics. This opens the door for memory offloading to target not just cold pages but also warm ones. Meta is concentrating its near-term architecture work on leveraging CXL devices as a viable offload back end in this expanded role.