Speeding up Linux disk encryption

Disk encryption without the tax

Encryption at rest is a baseline requirement for any service that handles customer data. At Cloudflare, with more than 200 data centres, there is no debate about whether to encrypt—only about how to do it without sacrificing performance. When profiling revealed that Linux disk encryption was costing us dearly on throughput and latency, we dug into the kernel's dm-crypt implementation and found that much of the overhead was not from cryptography at all, but from excessive queueing. By bypassing those queues, we more than doubled encrypted disk throughput and cut IO latency in half.

The layering problem

Data encryption can be applied at different layers of the OS storage stack, mirroring how TLS and IPsec operate at different network layers. Higher-layer approaches like application-level encryption give developers maximum flexibility but demand cryptographic expertise and bypass the Linux page cache, forcing repeated decryption or custom caching logic. File-system-level encryption is transparent to applications and can be scoped to directories or per-file keys, but it leaves metadata—file sizes, directory structure, file counts—exposed.

Block-level encryption (full disk encryption) sits at the bottom of the software stack and protects everything, including metadata and free space. The trade-off is coarser control: one key for the whole disk, no per-file or per-user policies. For server deployments managed by a central platform team, that simplicity and comprehensive coverage wins. We encrypt everything by default—no sorting data into "important" and "not important" buckets—so the selective flexibility of higher layers is unnecessary.

Hardware-based full disk encryption exists, and some drives implement it in firmware. But proprietary firmware gets less security review than open software, and past implementations have had serious flaws. Microsoft has shifted toward software encryption for similar reasons. We standardise on the Linux kernel's dm-crypt, which is open source and widely audited.

Benchmarking the pain

To isolate encryption overhead from storage hardware noise, we benchmarked on a RAM disk—there is no persistent storage to introduce variability. We set up a 4GB ramdisk and created a LUKS-encrypted device mapper target on top of it, storing the LUKS header in a separate file so benchmarking writes would not corrupt the encrypted key. No filesystem was created on either device, keeping the test focused purely on block-level encryption.

$ sudo modprobe brd rd_nr=1 rd_size=4194304
$ ls /dev/ram0

Using the Flexible I/O tester (fio) with sequential 4K reads and writes, the unencrypted ramdisk delivered roughly 1126 MB/s in both directions. Adding dm-crypt on top collapsed that to around 147 MB/s—a drop of more than seven times.

$ sudo fio --filename=/dev/mapper/encrypted-ram0 --readwrite=readwrite --bs=4k --direct=1 --loops=1000000 --name=crypt
crypt: (g=0): rw=rw, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=1
fio-2.16
Starting 1 process
...
Run status group 0 (all jobs):
   READ: io=1693.7MB, aggrb=150874KB/s, minb=150874KB/s, maxb=150874KB/s, mint=11491msec, maxt=11491msec
  WRITE: io=1696.4MB, aggrb=151170KB/s, minb=151170KB/s, maxb=151170KB/s, mint=11491msec, maxt=11491msec

This was surprising because cryptsetup benchmark showed AES-XTS with a 256-bit key to be the fastest cipher available on the system—and that is what we were using. Even a rough back-of-the-envelope estimate accounting for CPU encryption throughput suggested we should have seen around 696 MB/s, not 294 MB/s. Something else was wrong.

Eliminating the obvious suspects

The cryptsetup tool exposes performance-related flags that seemed promising. The --perf-same_cpu_crypt flag, which performs encryption on the same CPU that submitted the IO request, made things slightly worse. --perf-submit_from_crypt_cpus, which offloads request submission to the encryption threads, improved throughput marginally to ~166 MB/s. Neither flag came close to closing the gap, and the dm-crypt mailing list was not sympathetic: "If the numbers disturb you, then this is from lack of understanding on your side."

We were not convinced that modern AES-NI hardware acceleration was the bottleneck. Our own research on TLS performance had already shown that cryptography on modern hardware is inexpensive, even at massive scale. The problem had to be architectural.

Following the request path

Tracing the code path revealed that dm-crypt is far from a simple proxy that encrypts buffers as they pass through. Write requests can be queued up to four times; read requests up to three. Each queue introduces scheduling latency and potential for contention.

When a filesystem issues a write, dm-crypt does not process it inline. The request goes into a workqueue named kcryptd, is later passed to the Linux Crypto API—which may itself process asynchronously—and then possibly gets sorted into a red-black tree before a separate kernel thread finally submits it down the stack. Read requests follow a similar path through a different workqueue, kcryptd_io, before decryption is scheduled.

Each of these queues was added for a sensible reason at the time. The original use of a workqueue for decryption dates to 2005, when performing decryption in interrupt context was genuinely unwise. Later changes added queueing to reduce kernel stack usage and prevent request starvation under memory pressure. In 2015, write sorting was introduced because encryption completion order could destroy sequential write patterns and degrade spinning disk performance.

Hardware has changed dramatically since those decisions were made. NVMe SSDs do not benefit from these sorting mechanisms the way spinning disks did. Kernel stacks have been expanded, and the CFQ IO scheduler that motivated some of the write sorting has been removed from the kernel entirely. The architecture was tuned for a generation of hardware that is no longer dominant.

Removing the queues

Rather than deleting code, we added a new dm-crypt flag that bypasses the queues and threads when enabled. This allows runtime switching between old and new behaviour—useful for safe production rollouts and A/B testing under real load. The patch is available on Cloudflare's GitHub Linux repository.

Eliminating dm-crypt's internal queues was only half the battle. The Linux Crypto API can also be asynchronous, queuing requests to its own threads. We wanted a fully synchronous path. Examining available AES-XTS implementations on our systems showed two usable options: the generic C implementation, xts(ecb(aes-generic)), and the hardware-accelerated __xts-aes-aesni. The AES-NI version is substantially faster but marked as internal—available only to other Crypto API modules, not external users like dm-crypt.

There is a deeper complication with AES-NI: it uses CPU registers reserved for the FPU. The kernel does not always preserve these registers in interrupt context to avoid the expense of saving and restoring FPU state. Using AES-NI from dm-crypt, which can execute in interrupt context, risks corrupting other processes' data—the very concern that motivated the original queueing design.

Our solution was a new Crypto API module called xtsproxy. It is synchronous and does no cryptography of its own; it routes requests. When FPU use is safe in the current context, it forwards requests to the fast internal AES-NI implementation. Otherwise, it falls back to the generic C implementation.

The full setup process involved loading the new patches, restarting an IO workload in one terminal, then reconfiguring the encrypted device through dmsetup to use xtsproxy and enable the new dm-crypt bypass flag. A suspend/resume cycle on the encrypted device applied the changes live.

$ sudo dmsetup suspend encrypted-ram0 && sudo dmsetup resume encrypted-ram0

The payoff

With the queueing removed, total throughput jumped from roughly 294 MB/s to ~640 MB/s—more than double—and much closer to the theoretical estimate of ~696 MB/s. IO latency, measured by the await statistic from iostat, was cut in half as well.

Production numbers confirmed the synthetic benchmarks. Comparing 99th-percentile cache-hit response times across three server configurations—unencrypted disks, default encrypted disks, and encrypted disks with our patches—showed the patched implementation to be indistinguishable from having no encryption at all. The default Linux implementation, by contrast, introduced significant latency in worst-case scenarios. Encryption, with these changes, became effectively free for our workload.

What went upstream

Our main dm-crypt patch was merged into the mainline Linux kernel and is available since version 5.9. The upstream version differs slightly: it exposes two independent flags to bypass the read and write workqueues separately, offering finer control than our original single flag. Full details are in the official dm-crypt documentation.

The story here is not that Linux disk encryption is inherently slow—it is that the kernel's implementation carries legacy design decisions from an era of spinning disks, smaller kernel stacks, and synchronous-only cryptographic primitives. Understanding the architecture made it possible to recover most of the theoretical performance ceiling.

The current patches are tuned for our hardware and workload profile. Linux must run everywhere, from high-end servers to constrained IoT devices across multiple architectures, and a solution that works well in one context may not suit another. But for those running similar configurations, the runtime flag makes it simple to test whether the bypass helps—the patches have been running across Cloudflare's production network on five generations of hardware, so they can be considered stable.