Home/Security/The Linux Kernel Key Retention Service and why you should use it in yo

Security

The Linux Kernel Key Retention Service and why you should use it in your next application

Many leaks happen because of software bugs and security vulnerabilities. In this post we will learn how the Linux kernel can help protect cryptographic keys from a whole class of potential security vulnerabilities: memory access violations.

OI

Oxana, IgnatCloudflare

·November 28, 2022

The Linux Kernel Key Retention Service and why you should use it in your next application

Why Cryptographic Keys Leak

Cryptographic keys protect our data, but they are themselves just data stored in memory. That makes them vulnerable to the same class of bugs that accounts for roughly 70% of software vulnerabilities in major codebases: memory access violations. When a process reads or writes outside its intended boundaries, sensitive material such as private keys can be exposed through stack reuse, heap overflows, or use-after-free errors.

Consider a simple C program where a function stores a key on the stack, finishes, and is followed by a logging function that allocates its own buffer at the same address:

#include <stdio.h>
#include <stdint.h>

static void encrypt(void)
{
    uint8_t key[] = "hunter2";
    printf("encrypting with super secret key: %s\n", key);
}

static void log_completion(void)
{
    /* oh no, we forgot to init the msg */
    char msg[8];
    printf("not important, just fyi: %s\n", msg);
}

int main(void)
{
    encrypt();
    /* notify that we're done */
    log_completion();
    return 0;
}

Compile and run it:

$ gcc -o broken broken.c
$ ./broken 
encrypting with super secret key: hunter2
not important, just fyi: hunter2

The logger prints the old key material instead of the intended message. Two problems exist: the key was never securely wiped from memory, and the buggy logging function can address any memory within the process. The first problem can be fixed with code; the second is inherent to how processes use virtual memory.

Each process receives a contiguous block of virtual memory. It cannot touch another process's address space, but it can read or write any location within its own. When a function returns, its stack frame is simply marked free — the data remains. A later function may reuse those addresses and inherit leftover sensitive data. Format-string bugs in C functions such as printf() can similarly walk the stack and pull data from previous frames.

Safer languages reduce, but do not eliminate, the problem. Python checks many boundaries yet has its own heap-buffer-overflow and heap-use-after-free CVEs. Go offers an unsafe package that sidesteps normal checks. Even the most widely used cryptography library, OpenSSL, leaked private keys via Heartbleed in 2014.

Isolation as Mitigation

Since memory bugs are unavoidable in practice, the better strategy is to keep keys where a memory violation cannot reach them. The classic approach is process isolation: two separate processes, where one holds the key and performs cryptographic operations, and the other makes requests. The key material never crosses the communication channel.

This is the model behind ssh-agent, which performs private-key operations on behalf of clients without ever sending the key over its request channel. Cloudflare's Keyless SSL uses the same principle, keeping customer keys in an isolated environment away from Internet-facing connections.

The two-process model works but requires building, deploying, and maintaining two programs with a defined communication interface, authentication, and lifecycle management. An alternative is to delegate those duties to an entity that already exists, already runs in its own address space, and already enforces permissions: the Linux kernel.

The Linux Kernel Key Retention Service

The Linux Kernel Key Retention Service was originally designed for in-kernel consumers such as dm-crypt and ecryptfs, but is also available to userspace programs. It stores keys outside the process address space, exposes system calls as its interface, and associates permissions and ACLs with every key object.

The service models two entity types: keys and keyrings. A keyring is itself a special type of key. Think of keys as files and keyrings as directories: keyrings hold keys and other keyrings, but only keys contain actual cryptographic material.

Keys have types that determine permitted operations. user and logon keys can hold arbitrary data blobs, but logon keys cannot be read back into userspace, making them suitable exclusively for in-kernel services. For applications that want to delegate crypto operations, the asymmetric type is relevant: it holds a private key in the kernel and lets authorized processes request signing or decryption. Today only RSA keys are supported; ECDSA support is in progress.

Keyrings control key lifetime and shared access. When a keyring is destroyed, all keys linked only to it are securely destroyed as well. While custom keyrings can be created manually, the most useful are the "special" keyrings that the kernel manages implicitly.

Process, User, and Session Keyrings

Among the implicit keyring categories, two matter most for typical applications.

A user keyring is bound to a user ID and shared by all processes running under that UID. Any process can store or retrieve a key, but when the UID is removed, everything in that keyring is destroyed.

Process keyrings come in three flavors. A process keyring is private to one process and dies with it — regardless of how termination happens. Even a hard crash results in kernel-managed key destruction. A thread keyring extends that privacy to a single thread; a multithreaded server can hold keys for different TLS certificates and ensure one thread cannot use another's private key. A session keyring is available to the current process and all its children, surviving until the topmost process exits.

The session keyring is particularly suited to interactive shell usage. Commands run in subshells, so anything added to a process keyring is destroyed as soon as the command terminates. For example, adding a key to the process keyring from a command line:

$ keyctl add user mykey hunter2 @p
742524855

Tracing the kernel's key destruction function confirms the key was immediately freed:

…
Attaching 1 probe...
destroying key 742524855

The session keyring avoids this problem by persisting for the session and being shared with child processes. Keys are destroyed when the session ends, typically at logout.

Replacing ssh-agent with the Kernel

A practical demonstration: patch OpenSSH to source private keys from the kernel instead of an agent. The change is small and touches only the key-retrieval logic:

diff --git a/ssh-rsa.c b/ssh-rsa.c
index 6516ddc1..797739bb 100644
--- a/ssh-rsa.c
+++ b/ssh-rsa.c
@@ -26,6 +26,7 @@
 
 #include <stdarg.h>
 #include <string.h>
+#include <stdbool.h>
 
 #include "sshbuf.h"
 #include "compat.h"
@@ -63,6 +64,7 @@ ssh_rsa_cleanup(struct sshkey *k)
 {
 	RSA_free(k->rsa);
 	k->rsa = NULL;
+	k->serial = 0;
 }
 
 static int
@@ -220,9 +222,14 @@ ssh_rsa_deserialize_private(const char *ktype, struct sshbuf *b,
 	int r;
 	BIGNUM *rsa_n = NULL, *rsa_e = NULL, *rsa_d = NULL;
 	BIGNUM *rsa_iqmp = NULL, *rsa_p = NULL, *rsa_q = NULL;
+	bool is_keyring = (strncmp(ktype, "ssh-rsa-keyring", strlen("ssh-rsa-keyring")) == 0);
 
+	if (is_keyring) {
+		if ((r = ssh_rsa_deserialize_public(ktype, b, key)) != 0)
+			goto out;
+	}
 	/* Note: can't reuse ssh_rsa_deserialize_public: e, n vs. n, e */
-	if (!sshkey_is_cert(key)) {
+	else if (!sshkey_is_cert(key)) {
 		if ((r = sshbuf_get_bignum2(b, &rsa_n)) != 0 ||
 		    (r = sshbuf_get_bignum2(b, &rsa_e)) != 0)
 			goto out;
@@ -232,28 +239,46 @@ ssh_rsa_deserialize_private(const char *ktype, struct sshbuf *b,
 		}
 		rsa_n = rsa_e = NULL; /* transferred */
 	}
-	if ((r = sshbuf_get_bignum2(b, &rsa_d)) != 0 ||
-	    (r = sshbuf_get_bignum2(b, &rsa_iqmp)) != 0 ||
-	    (r = sshbuf_get_bignum2(b, &rsa_p)) != 0 ||
-	    (r = sshbuf_get_bignum2(b, &rsa_q)) != 0)
-		goto out;
-	if (!RSA_set0_key(key->rsa, NULL, NULL, rsa_d)) {
-		r = SSH_ERR_LIBCRYPTO_ERROR;
-		goto out;
-	}
-	rsa_d = NULL; /* transferred */
-	if (!RSA_set0_factors(key->rsa, rsa_p, rsa_q)) {
-		r = SSH_ERR_LIBCRYPTO_ERROR;
-		goto out;
-	}
-	rsa_p = rsa_q = NULL; /* transferred */
 	if ((r = sshkey_check_rsa_length(key, 0)) != 0)
 		goto out;
-	if ((r = ssh_rsa_complete_crt_parameters(key, rsa_iqmp)) != 0)
-		goto out;
-	if (RSA_blinding_on(key->rsa, NULL) != 1) {
-		r = SSH_ERR_LIBCRYPTO_ERROR;
-		goto out;
+
+	if (is_keyring) {
+		char *name;
+		size_t len;
+
+		if ((r = sshbuf_get_cstring(b, &name, &len)) != 0)
+			goto out;
+
+		key->serial = request_key("asymmetric", name, NULL, KEY_SPEC_PROCESS_KEYRING);
+		free(name);
+
+		if (key->serial == -1) {
+			key->serial = 0;
+			r = SSH_ERR_KEY_NOT_FOUND;
+			goto out;
+		}
+	} else {
+		if ((r = sshbuf_get_bignum2(b, &rsa_d)) != 0 ||
+			(r = sshbuf_get_bignum2(b, &rsa_iqmp)) != 0 ||
+			(r = sshbuf_get_bignum2(b, &rsa_p)) != 0 ||
+			(r = sshbuf_get_bignum2(b, &rsa_q)) != 0)
+			goto out;
+		if (!RSA_set0_key(key->rsa, NULL, NULL, rsa_d)) {
+			r = SSH_ERR_LIBCRYPTO_ERROR;
+			goto out;
+		}
+		rsa_d = NULL; /* transferred */
+		if (!RSA_set0_factors(key->rsa, rsa_p, rsa_q)) {
+			r = SSH_ERR_LIBCRYPTO_ERROR;
+			goto out;
+		}
+		rsa_p = rsa_q = NULL; /* transferred */
+		if ((r = ssh_rsa_complete_crt_parameters(key, rsa_iqmp)) != 0)
+			goto out;
+		if (RSA_blinding_on(key->rsa, NULL) != 1) {
+			r = SSH_ERR_LIBCRYPTO_ERROR;
+			goto out;
+		}
 	}
 	/* success */
 	r = 0;
@@ -333,6 +358,21 @@ rsa_hash_alg_nid(int type)
 	}
 }
 
+static const char *
+rsa_hash_alg_keyctl_info(int type)
+{
+	switch (type) {
+	case SSH_DIGEST_SHA1:
+		return "enc=pkcs1 hash=sha1";
+	case SSH_DIGEST_SHA256:
+		return "enc=pkcs1 hash=sha256";
+	case SSH_DIGEST_SHA512:
+		return "enc=pkcs1 hash=sha512";
+	default:
+		return NULL;
+	}
+}
+
 int
 ssh_rsa_complete_crt_parameters(struct sshkey *key, const BIGNUM *iqmp)
 {
@@ -433,7 +473,14 @@ ssh_rsa_sign(struct sshkey *key,
 		goto out;
 	}
 
-	if (RSA_sign(nid, digest, hlen, sig, &len, key->rsa) != 1) {
+	if (key->serial > 0) {
+		len = keyctl_pkey_sign(key->serial, rsa_hash_alg_keyctl_info(hash_alg), digest, hlen, sig, slen);
+		if ((long)len == -1) {
+			ret = SSH_ERR_LIBCRYPTO_ERROR;
+			goto out;
+		}
+	}
+	else if (RSA_sign(nid, digest, hlen, sig, &len, key->rsa) != 1) {
 		ret = SSH_ERR_LIBCRYPTO_ERROR;
 		goto out;
 	}
@@ -705,6 +752,18 @@ const struct sshkey_impl sshkey_rsa_impl = {
 	/* .funcs = */		&sshkey_rsa_funcs,
 };
 
+const struct sshkey_impl sshkey_rsa_keyring_impl = {
+	/* .name = */		"ssh-rsa-keyring",
+	/* .shortname = */	"RSA",
+	/* .sigalg = */		NULL,
+	/* .type = */		KEY_RSA,
+	/* .nid = */		0,
+	/* .cert = */		0,
+	/* .sigonly = */	0,
+	/* .keybits = */	0,
+	/* .funcs = */		&sshkey_rsa_funcs,
+};
+
 const struct sshkey_impl sshkey_rsa_cert_impl = {
 	/* .name = */		"[email protected]",
 	/* .shortname = */	"RSA-CERT",
diff --git a/sshkey.c b/sshkey.c
index 43712253..3524ad37 100644
--- a/sshkey.c
+++ b/sshkey.c
@@ -115,6 +115,7 @@ extern const struct sshkey_impl sshkey_ecdsa_nistp521_cert_impl;
 #  endif /* OPENSSL_HAS_NISTP521 */
 # endif /* OPENSSL_HAS_ECC */
 extern const struct sshkey_impl sshkey_rsa_impl;
+extern const struct sshkey_impl sshkey_rsa_keyring_impl;
 extern const struct sshkey_impl sshkey_rsa_cert_impl;
 extern const struct sshkey_impl sshkey_rsa_sha256_impl;
 extern const struct sshkey_impl sshkey_rsa_sha256_cert_impl;
@@ -154,6 +155,7 @@ const struct sshkey_impl * const keyimpls[] = {
 	&sshkey_dss_impl,
 	&sshkey_dsa_cert_impl,
 	&sshkey_rsa_impl,
+	&sshkey_rsa_keyring_impl,
 	&sshkey_rsa_cert_impl,
 	&sshkey_rsa_sha256_impl,
 	&sshkey_rsa_sha256_cert_impl,
diff --git a/sshkey.h b/sshkey.h
index 771c4bce..a7ae45f6 100644
--- a/sshkey.h
+++ b/sshkey.h
@@ -29,6 +29,7 @@
 #include <sys/types.h>
 
 #ifdef WITH_OPENSSL
+#include <keyutils.h>
 #include <openssl/rsa.h>
 #include <openssl/dsa.h>
 # ifdef OPENSSL_HAS_ECC
@@ -153,6 +154,7 @@ struct sshkey {
 	size_t	shielded_len;
 	u_char	*shield_prekey;
 	size_t	shield_prekey_len;
+	key_serial_t serial;
 };
 
 #define	ED25519_SK_SZ	crypto_sign_ed25519_SECRETKEYBYTES

The patch applies to the current OpenSSH git tree. Build it, linking libkeyutils for convenience wrappers around the keyring syscalls. PKCS11 support is disabled to avoid a symbol naming conflict with libkeyutils.

$ autoreconf
$ ./configure --with-libs=-lkeyutils --disable-pkcs11
…
$ make
…

Generate an RSA key in PKCS8 format, which the kernel requires (OpenSSH's default format is not accepted):

$ ./ssh-keygen -b 4096 -m PKCS8
Generating public/private rsa key pair.
…

An ssh-add-keyring.sh script loads the key into the current session keyring under the name myssh. It also creates a pseudo-key file at ~/.ssh/id_rsa_keyring that contains only the key name in native OpenSSH format. The main ssh process needs this file to know which in-kernel key to request.

#/bin/bash -e

in=$1
key_desc=$2
keyring=$3

in_pub=$in.pub
key=$(mktemp)
out="${in}_keyring"

function finish {
    rm -rf $key
}
trap finish EXIT

# https://github.com/openssh/openssh-portable/blob/master/PROTOCOL.key
# null-terminanted openssh-key-v1
printf 'openssh-key-v1\0' > $key
# cipher: none
echo '00000004' | xxd -r -p >> $key
echo -n 'none' >> $key
# kdf: none
echo '00000004' | xxd -r -p >> $key
echo -n 'none' >> $key
# no kdf options
echo '00000000' | xxd -r -p >> $key
# one key in the blob
echo '00000001' | xxd -r -p >> $key

# grab the hex public key without the (00000007 || ssh-rsa) preamble
pub_key=$(awk '{ print $2 }' $in_pub | base64 -d | xxd -s 11 -p | tr -d '\n')
# size of the following public key with the (0000000f || ssh-rsa-keyring) preamble
printf '%08x' $(( ${#pub_key} / 2 + 19 )) | xxd -r -p >> $key
# preamble for the public key
# ssh-rsa-keyring in prepended with length of the string
echo '0000000f' | xxd -r -p >> $key
echo -n 'ssh-rsa-keyring' >> $key
# the public key itself
echo $pub_key | xxd -r -p >> $key

# the private key is just a key description in the Linux keyring
# ssh will use it to actually find the corresponding key serial
# grab the comment from the public key
comment=$(awk '{ print $3 }' $in_pub)
# so the total size of the private key is
# two times the same 4 byte int +
# (0000000f || ssh-rsa-keyring) preamble +
# a copy of the public key (without preamble) +
# (size || key_desc) +
# (size || comment )
priv_sz=$(( 8 + 19 + ${#pub_key} / 2 + 4 + ${#key_desc} + 4 + ${#comment} ))
# we need to pad the size to 8 bytes
pad=$(( 8 - $(( priv_sz % 8 )) ))
# so, total private key size
printf '%08x' $(( $priv_sz + $pad )) | xxd -r -p >> $key
# repeated 4-byte int
echo '0102030401020304' | xxd -r -p >> $key
# preamble for the private key
echo '0000000f' | xxd -r -p >> $key
echo -n 'ssh-rsa-keyring' >> $key
# public key
echo $pub_key | xxd -r -p >> $key
# private key description in the keyring
printf '%08x' ${#key_desc} | xxd -r -p >> $key
echo -n $key_desc >> $key
# comment
printf '%08x' ${#comment} | xxd -r -p >> $key
echo -n $comment >> $key
# padding
for (( i = 1; i <= $pad; i++ )); do
    echo 0$i | xxd -r -p >> $key
done

echo '-----BEGIN OPENSSH PRIVATE KEY-----' > $out
base64 $key >> $out
echo '-----END OPENSSH PRIVATE KEY-----' >> $out
chmod 600 $out

# load the PKCS8 private key into the designated keyring
openssl pkcs8 -in $in -topk8 -outform DER -nocrypt | keyctl padd asymmetric $key_desc $keyring

After ensuring the SSH server accepts the generated key, sign in with SSH_AUTH_SOCK unset so no agent is consulted:

$ SSH_AUTH_SOCK="" ./ssh -i ~/.ssh/id_rsa_keyring localhost
The authenticity of host 'localhost (::1)' can't be established.
ED25519 key fingerprint is SHA256:3zk7Z3i9qZZrSdHvBp2aUYtxHACmZNeLLEqsXltynAY.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Linux dev 5.15.79-cloudflare-2022.11.6 #1 SMP Mon Sep 27 00:00:00 UTC 2010 x86_64
…

The operation succeeds without any password prompt. The private key never entered the SSH client process; the kernel performed the signing operation.

Choosing the Right Keyring

The session keyring used in this example has a lifespan tied to the login session. Logging out and back in destroys the key, forcing the script to be rerun. A second terminal during the same session will not see the key either, because each login session has its own session keyring instance:

$ keyctl show
Session Keyring
 333158329 --alswrv   1000  1000  keyring: _ses
 846694921 --alswrv   1000 65534   \_ keyring: _uid.1000

Switching the script to use the user keyring (@u instead of @s) makes the key accessible from all processes under the same UID, across logins. The tradeoff is that any process running as that user can use the key.

The Linux Kernel Key Retention Service turns the kernel into a secure, always-available key store with fine-grained lifecycle control. For applications that handle private keys, moving the keys out of the process address space and into the kernel removes an entire class of leak scenarios, at the cost of a small patch on the application side.

OI

Written by

Oxana, Ignat

Cloudflare

Writes for Cloudflare. Part of the Tech Report engineering index.

All articles by Oxana, Ignat →