Why a CDN might keep unpopular assets in memory
Cloudflare's CDN cache is backed by high-IOPS NVMe SSDs that serve millions of hits per second. While these drives are fast and reliable, cache hit tail latency is dominated by SSD IO capacity. Flash memory also wears out from write operations, and the cost of replacing dead drives—including hardware, shipping, labor, and downtime—is a non-trivial operational expense.
We developed a memory-SSD hybrid storage system that deliberately places unpopular assets in memory. The result is lower hit tail latency and reduced SSD wear for all customers.
This sounds backwards. Common sense says memory should hold the most popular assets since it's faster than SSDs. Linux's page cache already handles that. The real opportunity lies elsewhere: keeping unpopular assets out of the write path entirely, and using memory as a staging area so that only assets that become popular ever reach the disk.
How the page cache already helps—and where it falls short
Linux organizes files into pages and uses available system memory as a page cache. On a typical Cloudflare edge server with 256GB of physical memory, services consume about 87.71GB of RSS (with our cache service using 4.1GB), while the page cache totals 128.6GB, with 41.6GB serving cached web assets.
The page cache uses an LRU-based algorithm to keep frequently read pages in memory, so popular assets are already served from RAM. It also buffers writes, reducing repetitive writes to the same file. But in a CDN workload, most assets are static and written to disk once. Unlike database workloads with frequent in-place updates, there are very few repeated writes to a single asset; every page eventually gets flushed to disk. So the write-buffering benefit doesn't help much.
Our cache system knows more than the kernel: it sees content type, access frequency, and TTL values. That context lets us outperform the page cache—not for popular assets, which Linux already handles well, but for unpopular ones.
Why unpopular assets are the problem
SSD wear is caused entirely by writes, not reads. With data centers in 200 cities worldwide, the cost of shipping and replacing failed drives can rival the drive cost itself. Writes also slow down reads: program/erase (P/E) operations on flash chips block read operations to the same chips. The more writes, the slower the reads, and since popular assets are already in the page cache, this latency impact hits our tail latency hardest.
Many cached assets are "one-hit-wonders"—accessed once, cached, and never read again. Some are evicted before a single read because they never became popular. Not caching them in the first place wouldn't hurt website performance at all.
To quantify this, we ran an experiment: using a representative traffic sample, we modified caching logic to only cache an asset the second time the server encountered it.
The red line marks the experiment start; the green line is the experimental group, the yellow line the control. Disk writes dropped by roughly half, and disk hit tail latency improved by approximately five percent. An additional, less obvious benefit: avoiding one-hit-wonders increases effective cache capacity by removing competing pressure, which raises cache hit ratio and retention.
Two approaches, one clear winner
One idea is to remember asset appearances but not cache them on the first few misses. Data structures like hash tables, Bloom filters, Counting Bloom filters, or Count-min sketches can track appearances with various trade-offs. But this approach forces every asset to miss at least twice before being cached, amplifying cache misses, bandwidth costs, and server utilization. That's unacceptable.
A better design: put every asset destined for disk into memory first, in what we call a transient cache. Assets are promoted to the SSD-backed permanent cache only after they are accessed enough times to signal popularity. Otherwise, they are evicted from the transient cache.
This ensures disk writes are spent only on popular assets while every asset is missed only once.
Production trade-offs and learnings
The transient cache doesn't create performance out of nothing—it trades other system resources for improved customer latency, and the trade-offs need careful tuning.
Memory footprint. The transient cache's size dictates how long an asset lives before eviction. If it's too small, new assets evict old ones before they receive enough hits to earn promotion to disk. That shortens retention and causes more misses, higher cost, and worse performance.
Competition with page cache. Memory used for unpopular assets in the transient cache is memory that can't hold popular assets in the page cache. Finding the sweet spot depends on traffic volume, usage patterns, and hardware.
Competition with process memory. Process RSS is a hard requirement, and some services temporarily double their memory during "zero downtime upgrades," running old and new versions side by side. Without enough physical memory headroom, reduced page cache space causes unacceptable IO pressure.
Given these constraints, we enabled the system conservatively at first: only on newer servers with more physical memory, and only on a subset of assets. By tuning transient cache size and the percentage of requests that use it, we're mapping the trade-off space between performance, retention, and memory use.
The chart below shows IO usage before and after (red line) enabling transient cache in production.
Disk writes dropped by 25% at peak and 20% off peak, which should extend SSD lifespan proportionally. More importantly for customers, CDN cache hit tail latency measurably decreased.
What’s Next for the Hybrid Cache
The transient cache described above is the first step toward a more intelligent memory-SSD storage hierarchy, but rolling it out broadly presents clear next steps and open questions.
Scaling the Deployment
Right now the transient cache is limited to certain hardware generations and a fraction of traffic. As older servers are retired in favor of machines with more physical memory, a larger share of requests can be routed through the cache. Early experiments indicate that applying it to all traffic in some data centers could cut disk writes by as much as 70% and reduce user-visible tail latency at peak hours by up to 20%.
Improving the Promotion Logic
The current promotion path from transient cache to durable storage relies on a simple hit-count heuristic. Adding more signals could make those decisions smarter. Asset TTL is an obvious candidate: refusing to promote an asset that is only seconds away from expiring would avoid unnecessary disk writes.
Demotion also deserves attention. Assets can be moved from the persistent cache back to the transient cache, either actively or as part of eviction, based on similar criteria. Demotion alone does not reduce writes to the persistent cache, but a well-timed demotion can improve overall hit ratio and cache retention.
Extending the Cache Hierarchy
Memory was chosen for the transient cache because it has essentially no wear-out cost—a property it shares with HDDs. That opens the door to a three-tier hybrid spanning memory, SSDs, and HDDs, where each tier’s cost and performance profile is matched to the workload it serves.
Conclusion
Caching unpopular assets in memory is a deliberate trade-off: it consumes system memory in exchange for longer SSD life and lower tail latency. Under the current configuration, the approach delivers both benefits without degrading performance. The transient cache also lays groundwork for better eviction heuristics and a richer storage hierarchy.
Whether this strategy fits elsewhere depends heavily on workload characteristics and hardware constraints. But the core idea—questioning which assets belong in your fastest tier—may inspire other unconventional approaches to building faster, more efficient systems.
