Hardening Cloudflare Pages: Lessons from a Bug Bounty Collaboration

Cloudflare Pages recently went through a security review with researchers from Assetnote through our Public Bug Bounty program. The collaboration uncovered several vulnerabilities in the Pages build pipeline, all of which have been patched. There is no outstanding risk to Pages customers. The researchers' full write-up is available on their blog.

The core challenge Pages faced was running untrusted code safely in a multi-tenant CI/CD environment. This is a notoriously difficult problem, and the findings from this research showed that our initial architecture provided too large an attack surface. As a result, we completely re-architected the platform to use gVisor for stronger build isolation.

How vulnerability reports are handled

Reports submitted through HackerOne follow a defined process:

  1. The issue is investigated to understand its criticality.
  2. Engineering teams scope, implement, and validate a fix. Urgent issues get immediate attention; lower-severity ones are tracked alongside normal bug-fixing cadences.
  3. Our Detection and Response team reviews high-severity issues for signs of prior exploitation.

For most of the bugs below, audit logs gave us definitive signals that only the trusted researchers had exploited them. For one issue, we could not meaningfully prove that this was the case, so we took the precaution of notifying all potentially affected Pages users.

Command injection in build initialization

The first class of bugs involved command injection during build setup. A flaw in input validation made it possible to execute arbitrary code by controlling the root_dir parameter in a build's repository-cloning step. The attacker could craft a malicious root_dir value to dump the build process's environment variables to a file. Those variables included our GitHub bot's authorization key, which would have exposed the repositories — many of them private — of other Pages customers.

The fix involved tightening input validation for that field and rolling the disclosed keys. Audit logs confirmed this attack had never been performed before the researchers reported it.

An almost identical injection point existed in the asset-publishing step. We applied the same fixes — input validation and secret rotation — and again verified via logs that no one else had used the exposed credentials.

Privilege escalation through file permissions

During the build process, a program called /opt/pages/bin/pages-metadata-generator is invoked. This binary had Linux permissions of 777, meaning any user on the system could overwrite it. Since it runs with access to a full set of environment variables, an attacker could replace it with their own code before the next build ran.

#!/bin/bash
cp pages-metadata-generator /opt/pages/bin/pages-metadata-generator

The attackers' proof of concept was a simple reverse shell, sufficient to read the environment variables the process was invoked with. We fixed the file permissions, rotated the credentials, and verified in audit logs that nothing outside the research had touched them.

A variation on this theme was a PATH injection. The build environment's PATH variable included a large set of directories for compatibility with different developer tools, but some of these directories had incorrect filesystem permissions. This allowed a malicious version of bash to be placed in a PATH directory and later executed by the build process. Again, the fix was patching the permissions, rotating credentials, and validating via logs.

Container escapes in the old architecture

At the time of this research, Cloudflare Pages builds ran on Azure Pipelines inside highly privileged containers with the Docker socket mounted. Once the researchers got root in a container, escaping was straightforward: install Docker, mount the host's root directory, and recover Azure DevOps credentials from the host. Those credentials granted access to the Azure Organization running Pages.

The recovered credentials pointed to highly audited APIs, which let us confirm there had been no prior exploitation.

Re-architecture and a new issue

These findings drove us to migrate Pages builds from Azure to Kubernetes. The new design gives each build a Pod seeded with only the minimum set of credentials needed, which shrinks the attack surface significantly.

However, during this migration, a critical iptables rule was left off the Kubernetes control plane. This omission made it easy to curl the Kubernetes API and read secrets belonging to other Pods — each representing a separate Pages build. One of the exposed secrets was the GitHub OAuth secret, which would have allowed reading other customers' repositories.

We patched this quickly by adding iptables rules to block network connections to the control plane. In this case, unlike the prior bugs, we did not have the logs necessary to rule out exploitation by someone other than the researchers. So we notified all Pages customers who had ever had a build on the Kubernetes-based infrastructure, emailing them on February 3 to explain the situation.

Key takeaways

CI/CD security is a hard problem. Bug bounties that keep experienced researchers engaged are a valuable tool for stress-testing that infrastructure. While listing multiple vulnerabilities might seem like bad news, the depth and quality of the research gives us real confidence in the security of the resulting architecture — the weaknesses that existed are now fixed, and the new design is stronger for having been tested so thoroughly.

Security researchers interested in working with us can find details at hackerone.com/cloudflare.