Why a Cloudflare service was eating 30GB of RAM

When Cloudflare migrated the storage engine for its Quicksilver configuration distribution system to RocksDB, the service's memory footprint grew far beyond what the workload should have required. After startup, memory usage sat around 15GB and then climbed steadily for days until it leveled off near 30GB. Heap profiling ruled out a leak: the application's actual heap usage was roughly a third of the RSS value reported by the operating system. The gap pointed not at the application, but at the memory allocator.

Testing the service with TCMalloc on a few instances confirmed the suspicion — memory consumption dropped dramatically. The fix was simply swapping the default glibc allocator for one designed for multithreaded workloads. Understanding why that swap mattered requires a look at how the two allocators handle memory internally.

How glibc malloc fragments memory

glibc's malloc manages memory in contiguous regions called arenas. Its design assumes that applications free memory roughly in the reverse order of allocation — when that assumption breaks, chunks of free memory get stuck behind still-allocated objects and can't be returned to the operating system. malloc can only release memory from the top of the heap, so a single small live allocation can pin down a large block of otherwise free memory below it.

That's the essence of fragmentation: memory that the application no longer uses remains reserved by the process and unavailable to other services. All allocators face fragmentation, but the severity depends heavily on their design.

glibc malloc's approach to multithreading makes the problem worse. To avoid lock contention on a single arena, malloc creates multiple arenas and spreads thread allocations across them. When a thread can't acquire the lock on its preferred arena, it tries the next one, and if all are locked it creates a new arena — up to eight per core. Cloudflare's service runs about 25 threads and saw 60–80 arenas created under this scheme.

Each arena is fully independent; memory held in one arena can never be reused by another. In steady state, this leads to a situation where one thread allocates from an arena that lacks a suitable free chunk while another arena holds that exact chunk size idle. The process grows because fragmentation is computed per-arena rather than across the whole address space.

glibc malloc does provide malloc_trim() to walk arenas and release unused chunks, but it must be called explicitly by the application. Contrary to a long-standing man page bug — finally fixed after more than 15 years — free() does not trigger trimming automatically. Common tuning options exist, but each has a cost:

  • Lowering MALLOC_ARENA_MAX (usually to 2) reduces the number of arenas and improves memory reuse, at the expense of increased lock contention.
  • Calling malloc_trim() periodically forces locked chunks back to the system, but requires locking each arena and triggers syscalls and subsequent page faults.
  • Raising M_MMAP_THRESHOLD routes large allocations through mmap(), which avoids arena fragmentation entirely but adds an expensive syscall per allocation and is capped by a system limit on the number of mmap'd chunks.

The net effect: glibc malloc's per-arena design trades memory efficiency for lock-free concurrency — and the trade can cost two to three times the memory a workload actually needs.

TCMalloc's three-tier architecture

TCMalloc was designed for multithreading from the start. Its architecture separates memory management into three tiers:

  • The back-end acquires large chunks from the operating system, serves large allocation requests, and returns unused chunks to the OS.
  • The front-end maintains a per-core cache to serve allocation requests quickly.
  • The middle-end sits between the two: it refills caches, collects unused memory, and — critically — can move memory from one cache to another.

That last capability is what distinguishes TCMalloc from glibc malloc. In the earlier fragmentation scenario — where a thread requests 20KB and finds the chunk available only in another arena — TCMalloc solves the problem directly. A cache holding an unused chunk returns it to the middle-end; when another cache needs that size, it pulls the memory from the middle-end instead of growing the process footprint.

Memory freed by one thread can therefore be reused by any other thread, dramatically improving memory utilization in multithreaded services.

Allocator choice matters

After deploying TCMalloc, Cloudflare cut Quicksilver's memory usage by roughly 2.5 times. The lesson isn't specific to RocksDB or Cloudflare's architecture — it applies to any long-running, multithreaded service where memory efficiency matters. Allocators built for multithreaded workloads, including TCMalloc and jemalloc, generally provide far better memory reuse than glibc malloc's arena design.

If your application runs for a long time, spawns many threads, and holds a large RSS, the allocator itself may be the source of the waste. It's worth measuring how much memory your service is actually using — versus how much it has reserved.