Keeping GCP Project-Wide SSH Keys From Overstaying Their Welcome

When a developer uses gcloud compute ssh to connect to a VM in a Google Cloud Platform project, the public key used for that session is written to the project's metadata. From that point on, the key can be used to reach any Linux VM in that project that doesn't explicitly block project-wide keys. There is no built-in expiration for these metadata entries, so keys accumulate indefinitely unless someone removes them by hand.

The Infrastructure Security team at Shopify set out to change that with a tool called SSH-Pruner. The goal was straightforward: keep project-wide SSH keys from persisting past the moment they're needed, without disrupting the developers who rely on SSH access for debugging Kubernetes or Docker workloads.

How we manage GCP project-wide SSH keys at Shopify. Image by Samantha Lam on Unsplash

Why Persistent Keys Are a Problem

The risk profile for a lingering SSH key is similar to leaving a door code active after a guest checks out. It is hard to know who used it, when, and whether they should have had access at all. If a private key falls into the wrong hands, an old public key in project metadata can become an undetected entry point. Google Compute Engine's OS Login feature, which ties access to IAM identities, could mitigate this, but it doesn't work with Google Kubernetes Engine, so Shopify had to find another path.

An early attempt at solving this simply swept every SSH key out of every project with compute enabled. That approach had three flaws:

  • It deleted Google's own infrastructure management keys along with employee keys.
  • Scripts that expected a key to persist for a while could fail mid-execution.
  • Re-adding keys via project metadata is slow, and scaling that operation across many keys and nodes made it prohibitive.

The team needed a solution that met a stricter set of requirements: keys should not outlive their usefulness; developer workflows should see minimal impact; Shopify-controlled keys must be distinguished from Google's; and the tool needed to produce logging that explains why and when keys appear.

The Expiration Approach That Didn't Work

The obvious answer was to attach an expiration date to keys that lacked one. SSH-Pruner would prune expired keys and tag valid keys with an expireOn field. When run on a compute-enabled project, the tool would parse existing keys, drop invalid or expired ones, add expireOn where missing, and overwrite the project's metadata with the cleaned set.

It looked good until testing on a non-production project revealed a fatal flaw. The team connected to a VM twice—once with an expireOn and once without—and confirmed the metadata contained both variations. After running SSH-Pruner, the key with no expiration was removed as intended, and the other remained. But the next time a connection was made without specifying expireOn, a duplicate of that key was re-added to the metadata without an expiration date. Any developer using the standard gcloud compute ssh path would regenerate a permanent key on the next connection, and the pruning loop would start all over again.

Redesigning the Pruner

Given that expiration tags alone couldn't hold the line, the team went back to the drawing board. The revised approach preserves the spirit of the original attempt—periodic removal of stale keys—but avoids the collateral damage. The new logic:

  • Fetches all projects for the organization via the Google Cloud APIs.
  • Filters for projects where compute.googleapis.com is enabled.
  • Reads the project-level metadata, skipping entries that aren't ssh-keys (Shopify keys) or sshKeys (Google's keys).
  • Parses each metadata line into a structured record.
  • Rebuilds the metadata object with only unexpired keys before writing it back to the project.
  • Logs each step in a readable format for auditing and trend analysis.

The tool also needs to recognize and preserve Google's own keys, which live in the sshKeys metadata field. Shopify's keys belong in ssh-keys; anything found where it shouldn't be gets removed.

One further adjustment was required. The initial plan to add expireOn to keys without it was abandoned because it didn't stop the regeneration cycle. Instead, SSH-Pruner removes any key that does not carry a valid expiration date. The trade-off is that developers who create keys without specifying an expiration will need to generate a new one on their next connection—an acceptable disruption compared to letting keys persist indefinitely.

Caution: Managing SSH keys in metadata is only for advanced users who are unable to use other tools such as OS Login to manually manage SSH keys. If you manage SSH keys in metadata yourself, you risk disrupting the ability of your project members to connect to instances. Additionally, you risk allowing your instance to be accessed by users who aren't part of your project. For more information, see risks of manual key management.

Rolling It Out

Rather than reworking the prior codebase, Shopify built SSH-Pruner from scratch as a Go application. This was a deliberate choice, giving the team full confidence in each code path and ensuring compatibility with current Google API versions. It also gave the owning engineer room to learn the language and the domain deeply.

The tool is already running as a cron job on selected projects. Full rollout to the rest of the organization will be gradual, with careful observation of key removal frequency so that no team finds its access unexpectedly revoked mid-task. The balance the team is looking for is simple to state but hard to achieve: the shortest possible lifespan for access that still leaves developers unimpeded.

Why Persistent Keys Need Proactive Attention

Working through this project reinforced a familiar lesson: test early and often. It’s far better to catch flaws in a proposed solution before you’ve invested significant time in it.

When the systems you manage hold sensitive data or control access to important resources, security precautions aren’t optional—they’re a baseline requirement. Even when a problem looks simple, the fix can quickly become complex. In those cases, it’s worth erring on the side of being overprotective.

Security vulnerabilities rarely announce themselves. They sit dormant until someone finds a way to exploit them. Persistent SSH keys are a prime example: the risk can seem low enough that addressing them feels like busywork. But waiting for an incident to justify the effort is the wrong approach. The goal should be to identify and close gaps before they become problems, not just to react when something breaks.

That’s the mindset that drives meaningful security work. Small, focused efforts—like cleaning up project-wide SSH key management—can have an outsized impact. It’s the kind of simple but effective project that often gets overlooked until it’s too late.