Why Encrypt Memory?

Encryption typically addresses data in transit and data at rest. But what about data in use — specifically, the contents of RAM? In most systems, memory is stored in the clear, leaving it vulnerable to physical attacks. As persistent memory technologies become more common, the attack surface expands: DRAM interface snooping, hardware devices that access host memory, freezing and stealing DIMMs, and outright theft of NVDIMMs are all realistic threat vectors.

Trusted Execution Environments (enclaves) offer some protection by isolating software execution at runtime, but they were designed for small pieces of code, not full systems. Running an entire OS inside an enclave comes with significant limitations and performance penalties. Enclaves also don't address physical attacks against the memory itself.

AMD's Approach: SME and TSME

AMD's EPYC processor line includes Secure Memory Encryption (SME), an x86 instruction set extension that can mark individual memory pages as encrypted using standard page tables. When a page is marked encrypted, the memory controller automatically decrypts it on reads from DRAM and encrypts it on writes. This protects DRAM contents from physical attacks on the system.

SME consists of two key components:

  • AES-128 encryption engine: Embedded in the memory controller, responsible for encrypting and decrypting main memory data when provided with an appropriate key.
  • AMD Secure Processor (AMD-SP): An on-die 32-bit ARM Cortex A5 CPU that handles cryptographic key generation and management, acting like a mini hardware security module.

Enabling standard SME requires setting the model specific register MSR 0xC001_0010[SMEE]. This activates the ability to use an encryption bit in page table entries:

  • 0 = memory encryption features are disabled
  • 1 = memory encryption features are enabled

Once enabled, a physical address bit (C-Bit) determines whether a page is encrypted. The operating system sets this bit in the page table entry (PTE) to 1 to mark a page as protected, and the AES engine in the memory controller handles the rest.

Securing Memory at EPYC Scale Embedded Image - BBtKex

Transparent SME: Full Memory Protection

Manually flagging individual pages offers flexibility, but for complete physical protection, Transparent SME (TSME) is the better option. In TSME mode, all memory is encrypted regardless of any per-page encryption bits — including instruction pages, data pages, and the page tables themselves.

Activating TSME is straightforward:

  1. Set the appropriate BIOS flag:
Securing Memory at EPYC Scale Embedded Image - 9JNL3J

2. Enable kernel support with:

CONFIG_AMD_MEM_ENCRYPT=y

After reboot, dmesg should confirm the feature is active:

$ sudo dmesg | grep SME
[    2.537160] AMD Secure Memory Encryption (SME) active

Measuring the Performance Impact

To evaluate whether the security benefit justified the potential performance cost, we ran tests on a server mirroring our production edge hardware:

  • Memory: 8 x 32GB 2933MHz
  • CPU: AMD 2nd Gen EPYC 7642 with SMT enabled, NPS4 mode
  • OS: Debian 9
  • Kernel: 5.4.12

Three tools were used to assess the impact: STREAM for memory bandwidth, cryptsetup for cryptographic throughput, and Benchmarky, an internally developed benchmarking tool.

Memory Bandwidth with STREAM

We ran a custom STREAM binary with 24 threads across all available cores to measure sustainable memory bandwidth in MB/s. The tool executes four synthetic computational kernels sequentially, with each kernel's output feeding the next, and reports the best observed rates.

Securing Memory at EPYC Scale Embedded Image - WXXPOA
Securing Memory at EPYC Scale Embedded Image - ISTJYc

The results showed performance variation between 2.6% and 4.2%, with a mean impact of 3.7%. This was lower than the expected drop of over 5%.

Cryptographic Throughput with Cryptsetup

Though cryptsetup is typically used for disk encryption, its benchmarking utility measures host cryptographic performance by running key derivation functions in memory only.

$ sudo cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1      1162501 iterations per second for 256-bit key
PBKDF2-sha256    1403716 iterations per second for 256-bit key
PBKDF2-sha512    1161213 iterations per second for 256-bit key
PBKDF2-ripemd160  856679 iterations per second for 256-bit key
PBKDF2-whirlpool  661979 iterations per second for 256-bit key
Securing Memory at EPYC Scale Embedded Image - ZWZ1YL
Securing Memory at EPYC Scale Embedded Image - tGs9Nn

Application-Level Performance with Benchmarky

Benchmarky is a homegrown tool that runs synthetic workloads against a target to assess component performance. It leverages Cloudflare Workers to generate requests and collect response statistics, also reporting on stack component versions and their CPU usage. Each test runs 256 concurrent clients fetching a cached 10kB PNG image from a performance testing endpoint, measuring requests per second (RPS).

Securing Memory at EPYC Scale Embedded Image - itv2GD
Securing Memory at EPYC Scale Embedded Image - AnwkPM

A Small Price for Full Memory Encryption

Across all tests, the performance decrease was minimal — less than expected. AMD's own documentation acknowledges that memory encryption via the AES engine adds some latency to DRAM accesses, but the actual impact depends heavily on the workload. Averaging all 11 data points, the total performance drag was only 0.699%.

At production scale, enabling TSME significantly reduces the risk of data exfiltration from stolen servers. While awaiting industry-wide support for total memory encryption from other manufacturers, AMD's EPYC line has set a strong precedent for protecting next-generation edge hardware.