Analyzing Volatile Memory on a Google Kubernetes Engine Node

Forensic Memory Analysis on GKE Nodes

Spotify runs its production workloads on Google Kubernetes Engine (GKE) across five Google Cloud Platform regions, with hundreds of thousands of pods in more than 3,000 namespaces. When suspicious activity is detected, engineers need to analyze what happened quickly. While commercial monitoring solutions cover much of this ground, an in-house research project at Spotify uncovered a fully open source method for pulling a complete process and memory snapshot from a GKE node by combining three tools: AVML, dwarf2json, and Volatility 3.

This technique gives teams without commercial monitoring an open source alternative and provides a way to cross-check existing commercial solutions. The method works on any GKE node in production today.

Kubernetes Basics and the Kernel's Role

A GKE cluster follows the standard Kubernetes model: a control plane orchestrates worker nodes, and each node runs pods, which are groups of containers. Namespaces isolate groups of resources within a cluster. Understanding which processes are active in memory on a node means looking past the container abstraction to the underlying kernel.

The kernel is the layer between a node's operating system and its hardware. It handles process scheduling, memory management, file systems, device control, and networking. A memory dump of the kernel therefore reveals every process running on the node, regardless of which pod it belongs to.

Commercial solutions often use the extended Berkeley Packet Filter (eBPF) to safely access the kernel. That works, but it requires either purchasing a commercial eBPF-based solution or building a custom one. An alternative approach is to dump kernel memory and analyze it offline, which is what this research explored.

Three Steps to a Node Memory Snapshot

The workflow has three phases: create a kernel memory dump, build a symbol file for the kernel version, and analyze the dump against those symbols. The full integration was tested with Terraform and a Python script that calls the GCP API.

Step 1: Capture a Kernel Memory Dump

GKE nodes run Container-Optimized OS (COS), a hardened image that doesn't allow kernel modules. However, a privileged container can be added temporarily to gain access to /proc/kcore, the virtual file that represents physical memory. From there, the open source tool AVML can capture a memory dump.

This privileged container exists only for the duration of the capture and provides the necessary read access to kernel space for the AVML process.

Step 2: Build the Kernel Symbol File

Interpreting the raw memory dump requires an Intermediate Symbol File (ISF) specific to the node's kernel version. The source for this is vmlinux, the uncompressed kernel image. The tool dwarf2json converts the DWARF debug information in vmlinux into an ISF that Volatility 3 can consume.

The challenge was locating the correct vmlinux for a given COS version. Through research and conversations with Google engineers working on GKE and COS, an undocumented API was discovered. The file is available at https://storage.googleapis.com/cos-tools/$build_id/vmlinux, where $build_id is taken from the GKE node image name. For example, a node with build ID 16919.235.1 yields the file at:

https://storage.googleapis.com/cos-tools/16919.235.1/vmlinux

Step 3: Analyze the Dump with Volatility 3

With both the memory dump and the matching symbol file ready, Volatility 3 can map the raw dump back to running processes. In testing, this showed every process running on the node, including a test "attacker" pod that was running a Netcat listener, a command querying the local IP address, and a Python script. The output listed processes from the privileged capture pod and the test pod side by side, confirming that the method yields a complete view of node-level process activity.

An Open Source Complement to Commercial Tools

This combination of AVML, dwarf2json, and Volatility 3 gives Spotify, and any other GKE operator, a no-cost path to kernel memory forensics. The result is a point-in-time snapshot rather than continuous monitoring, so it fits naturally as a starting point for incident investigation or as a verification layer alongside commercial monitoring stacks.

The code from this research project is published on GitHub, and the work was presented at BSidesNYC 2023. Kubernetes is a registered trademark of the Linux Foundation in the United States and other countries.