Why the kernel is the last line of defense

Every process that runs on a Linux system — whether it's a web server, a container workload, or a management agent — executes under the authority of the kernel. The kernel mediates memory access, hardware I/O, and inter-process communication. That makes it the single most important enforcement point for any security policy. If an attacker can subvert the kernel, no amount of userspace hardening matters.

Modern Linux kernels expose several security mechanisms that can be tuned at build time and load time to limit the damage a compromised or malicious process can do. This article looks at some of the kernel security configurations Cloudflare adopts in production and why they matter.

Securing the path from firmware to kernel

Machine boot proceeds through several distinct stages, from firmware to bootloader to kernel to initramfs. With a secure boot architecture, each stage cryptographically verifies the integrity of the next before passing execution control. This creates a chain of trust where each verified component validates the next — provided the chain is unbroken from the very first firmware stage.

The Linux kernel is the endpoint of that chain's integrity guarantees in most systems. Without verifying that the kernel itself hasn't been tampered with, every subsequent security mechanism — SELinux, AppArmor, module signing, lockdown mode — rests on unverified code. This is why Cloudflare uses UEFI Secure Boot to anchor the kernel image, as described in detail previously.

The module loading problem

The boot process isn't the only place the kernel expands its code. Kernel modules can be loaded at any time during runtime: when new hardware is introduced, when additional network filtering features are required, or when a system administrator manually inserts a module. Unlike user space processes, kernel modules execute directly in the kernel's address space with no isolation between module code, module data, and core kernel subsystems.

That lack of separation makes uncontrolled module loading a fundamental threat to system integrity. A rogue module can reach into kernel memory anywhere. Consider a stock Debian 12 system with SELinux configured and enforcing.

The SELinux enforcement state is held in kernel memory. In the 6.1.76 kernel, the relevant structure is defined in security/selinux/include/security.h. The enforcing flag within that structure controls whether SELinux is actively enforcing policy. The structure carries a __randomize_layout attribute, which is intended to make the member layout unpredictable to potential exploit writers. However, the Debian kernel is compiled without CONFIG_SECURITY_SELINUX_DISABLE, and struct randomization is disabled in the Debian build because of the performance overhead it introduces.

Confirming the layout doesn't even require privileged access — kernel BTF information is available to any user. Using pahole, the enforcing field can be shown to live at the start of the selinux_state structure. With the struct layout known and the kernel's symbol table accessible, a simple kernel module can directly flip the enforcement flag in memory.

Such a module is straightforward to write. Load it into the running kernel, and SELinux is silently neutralized. Notably, this approach bypasses the normal setenforce 0 path which would emit audit messages through the selinuxfs interface. The kernel manipulation goes completely unnoticed by any monitoring that relies on those audit logs.

SELinux policy can sometimes block module insertion, but a module copied into the standard module path may bypass those restrictions depending on policy configuration. This demonstrates why monitoring module loads is a common recommendation in security standards — but monitoring is a reactive measure. A better approach is to prevent unsigned modules from loading at all.

Mandatory module signatures

Cloudflare's production kernel uses a different strategy. In its configuration, the enforcement of module signatures is required rather than optional. When Cloudflare's kernel attempts to insert a module that wasn't signed with the trusted key, the insertion fails with a Key was rejected by service error in the kernel log.

The distinction between mandatory and optional signature checking is critical. Stock Debian kernels support module signatures but don't enforce them — a module without a valid signature will still load, emitting a warning and tainting the kernel. That behavior leaves room for an attacker who can compile a module against the vendor kernel headers to slip it past the signature verifier and get it loaded.

Disposable signing keys

Mandatory module signatures raise an operational question: who holds the private signing key, and how is it protected? A stolen signing key defeats the entire scheme because anyone with it can sign and load a malicious module that the kernel will accept as legitimate.

Cloudflare avoids the key management burden with a straightforward technique: the kernel build system generates a fresh keypair for module signing when none is supplied. The public key gets embedded in the kernel image during build. The private key is used once — to sign every module built for that kernel — and then destroyed before the kernel is released to production.

The benefits are threefold:

  • Short key lifetime: each kernel release is coupled to a freshly generated, ephemeral signing key
  • Brevity of exposure: even if a build pipeline is compromised and a key is leaked, the key is obsolete once the next production kernel ships
  • No secrets in storage: there is no protected key repository to manage, rotate, or lose

The trade-off surfaces when a new driver is needed for a physically deployed kernel release — for instance, when a new piece of hardware arrives that requires support not present in the shipped module set. A fresh module for that deployed kernel can't be generated because the private key was deleted at compile time. In practice this is not a blocker for Cloudflare, which releases kernels roughly weekly and keeps up with the steady stream of bug fixes and vulnerability patches in the kernel. The public key for each kernel is also ephemeral, meaning that even a leaked build key cannot be used to sign code for a different kernel generation.

Module signature enforcement, paired with disposable signing keys, closed the class of kernel attacks that rely on inserting arbitrary code at runtime. The mandatory signature policy, however, becomes fully effective only when combined with the secure boot anchoring of the kernel itself. Together they form a loop: the trusted boot path leads to a kernel that only accepts additional code that is trusted via its own cryptographic chain back to the build time process.

Locking Down the Legacy kexec Interface

The kexec_load() system call predates modern system integrity expectations. It hands the kernel a collection of raw buffers containing code, plus an intermediate program called purgatory, and trusts that the kernel will execute it at the highest privilege level. As has been demonstrated publicly, this design can bypass secure boot and compromise system integrity — you can feed it code that disables SELinux or otherwise alters the running kernel. The original intent was to speed up kernel switches without a full reboot, but as Cloudflare found, the risk of improperly initialized hardware made it unattractive even before the security concerns were factored in.

The one valuable use case for kexec is crashdumping: when a production kernel crashes, a backup kernel loaded in advance can collect a memory dump for investigation. That capability remains useful, which is why the modern replacement, kexec_file_load(), is worth attention. Instead of accepting arbitrary buffers, it takes file descriptors for the kernel image and initrd and performs parsing inside the kernel, so only a valid kernel image can be loaded. It can also be configured to require proper signatures, ensuring only authorized code runs. A secure kexec configuration looks like this:

ignat@dev:~$ grep KEXEC /boot/config-`uname -r`
CONFIG_KEXEC_CORE=y
CONFIG_HAVE_IMA_KEXEC=y
# CONFIG_KEXEC is not set
CONFIG_KEXEC_FILE=y
CONFIG_KEXEC_SIG=y
CONFIG_KEXEC_SIG_FORCE=y
CONFIG_KEXEC_BZIMAGE_VERIFY_SIG=y
…

The legacy kexec_load() is disabled by turning off CONFIG_KEXEC, while crashdumping remains available via CONFIG_KEXEC_FILE=y with signature checks enforced through CONFIG_KEXEC_SIG=y and CONFIG_KEXEC_SIG_FORCE=y. Note that a stock Debian kernel still ships with the legacy system call enabled and without enforced signature checks for kexec_file_load(), similar to its module signature policy:

ignat@dev:~$ grep KEXEC /boot/config-6.1.0-18-cloud-amd64
CONFIG_KEXEC=y
CONFIG_KEXEC_FILE=y
CONFIG_ARCH_HAS_KEXEC_PURGATORY=y
CONFIG_KEXEC_SIG=y
# CONFIG_KEXEC_SIG_FORCE is not set
CONFIG_KEXEC_BZIMAGE_VERIFY_SIG=y
…

KASLR and Pointer Leak Protection

Kernel Address Space Layout Randomization (KASLR) shifts the kernel's code and data by a random offset on each boot. This is primarily a defense against targeted exploitation that relies on knowing the locations of internal structures. On a popular distribution kernel, anyone can download debug symbols and extract exact addresses from System.map, so a fixed layout makes crafting a malicious kernel module trivial. With KASLR, the same module will not hit its intended target — it will instead modify a random piece of kernel memory, likely causing a crash. Both the Cloudflare and Debian kernels enable KASLR:

ignat@dev:~$ grep RANDOMIZE_BASE /boot/config-`uname -r`
CONFIG_RANDOMIZE_BASE=y

KASLR alone is weak because everything shifts by a single offset. If the attacker learns one runtime kernel address, they can subtract the compile-time address of the same symbol (found via System.map) to recover the offset and thereby compute every other symbol's address. This is why modern kernels work to avoid leaking kernel addresses to unprivileged users in the first place.

A central tunable here is the kptr_restrict sysctl. Setting it to 1 keeps regular users from seeing kernel pointers:

ignat@dev:~$ sudo sysctl -w kernel.kptr_restrict=1
kernel.kptr_restrict = 1
ignat@dev:~$ grep selinux_state /proc/kallsyms
0000000000000000 B selinux_state

Privileged users retain visibility:

ignat@dev:~$ sudo grep selinux_state /proc/kallsyms
ffffffffb41bcae0 B selinux_state

Similarly, dmesg_restrict prevents unprivileged reads of the kernel log, which can leak pointers through various messages. While kptr_restrict must be set at each boot (or via a sysctl utility), dmesg_restrict can be enforced from boot time through CONFIG_SECURITY_DMESG_RESTRICT. Both the Cloudflare and Debian kernels use this configuration:

ignat@dev:~$ grep CONFIG_SECURITY_DMESG_RESTRICT /boot/config-`uname -r`
CONFIG_SECURITY_DMESG_RESTRICT=y

Neither /proc/kallsyms nor the kernel log is the only source of pointer leaks; the kernel contains plenty of legacy paths where new leaks are continuously discovered and patched. Staying current with kernel bugfix releases is an essential part of this defense.

The Lockdown LSM

Linux Security Modules (LSM) provide a hook-based framework for implementing Mandatory Access Control. Among the available modules, the Lockdown LSM targets exactly the kinds of integrity problems discussed above: unsigned module loading, unsigned kexec execution, and other operations that could compromise system integrity. It operates in three states, controlled via /sys/kernel/security/lockdown:

ignat@dev:~$ cat /sys/kernel/security/lockdown
[none] integrity confidentiality

In the none state, nothing is enforced. The integrity state blocks operations that could compromise the kernel's integrity, such as loading unsigned code. The confidentiality state goes further by attempting to block all information leakage from the kernel — but in practice this can be too restrictive for server workloads, as it disables runtime debugging facilities like perf and eBPF.

A stock Debian system starts in the none state:

ignat@dev:~$ uname -a
Linux dev 6.1.0-18-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux
ignat@dev:~$ cat /sys/kernel/security/lockdown
[none] integrity confidentiality

From there it is possible to move into integrity mode:

ignat@dev:~$ echo integrity | sudo tee /sys/kernel/security/lockdown
integrity
ignat@dev:~$ cat /sys/kernel/security/lockdown
none [integrity] confidentiality

Transition is one-way only: once in integrity, you can only advance to confidentiality, never back to none:

ignat@dev:~$ echo none | sudo tee /sys/kernel/security/lockdown
none
tee: /sys/kernel/security/lockdown: Operation not permitted

Even on a stock Debian kernel — which does not enforce module signatures by default — activating the Lockdown LSM in integrity mode prevents loading an unsigned, potentially malicious module:

ignat@dev:~$ sudo insmod mymod/mymod.ko
insmod: ERROR: could not insert module mymod/mymod.ko: Operation not permitted

The kernel log makes the cause clear:

ignat@dev:~$ sudo dmesg | tail -n 1
[21728.820129] Lockdown: insmod: unsigned module loading is restricted; see man kernel_lockdown.7

The Lockdown LSM thus tightens security on kernels that otherwise lack enforcing configuration, such as stock distributions. When compiling a custom kernel, you can set the initial Lockdown state to be more restrictive than none from the very start, which is what the Cloudflare production kernel does:

ignat@dev:~$ grep LOCK_DOWN /boot/config-6.6.17-cloudflare-2024.2.9
# CONFIG_LOCK_DOWN_KERNEL_FORCE_NONE is not set
CONFIG_LOCK_DOWN_KERNEL_FORCE_INTEGRITY=y
# CONFIG_LOCK_DOWN_KERNEL_FORCE_CONFIDENTIALITY is not set

A Practical Kernel Hardening Baseline

The configuration options discussed here — disabling legacy kexec_load() while retaining signed kexec_file_load(), enabling KASLR, restricting kernel pointer and log visibility, and enabling the Lockdown LSM in integrity mode — form a practical baseline for hardening Linux systems. They represent only a small subset of what the kernel community continues to develop and improve, but they address some of the most common vectors for compromising kernel integrity on stock and custom kernels alike.