How Cloudflare Contained an RCE in cdnjs' Auto-Update Pipeline

cdnjs is a free, open-source content delivery network that hosts JavaScript, CSS, images, and fonts for over 4,000 libraries. Websites use it to reduce load on their own origin servers, as files are delivered from Cloudflare's edge. A recent disclosure by researcher RyotaK detailed a remote code execution (RCE) vulnerability in the backend process that cdnjs uses to automatically keep its hosted libraries current. The flaw could have allowed an attacker to modify hosted assets.

The vulnerability is specific to the cdnjs platform and is not related to Cloudflare's core CDN infrastructure. Cloudflare uses the cdnjs project as a platform on top of its services. Following the report, Cloudflare's investigation confirmed that no existing libraries were tampered with — the researcher only published a new package to demonstrate the flaw.

Timeline of Events

On 2021-04-06, RyotaK published a malicious package to npm that exploited the vulnerability. At approximately 1129 GMT, the cdnjs auto-update process handled this package, which leaked credentials. GitHub's alerting systems caught the exposure and notified Cloudflare immediately. Cloudflare disabled the auto-update service and revoked all related credentials within an hour of the alert. The research team also formally reported the RCE through HackerOne.

Work on the initial patches, including a new version of the auto-update tool, was completed within 24 hours. Instead of stopping there, Cloudflare began a full redesign of the update pipeline, which was finished on 2021-06-03.

Immediate Response: Revoking Secrets and Halting the Service

Cloudflare's incident response began before the formal HackerOne submission. When GitHub alerted Cloudflare to the leaked credentials, an engineer immediately revoked all tokens and secrets. GitHub also automatically revoked the associated service token. As a second step, Cloudflare brought the vulnerable auto-update service fully offline. This halted exploitation but also prevented legitimate developers from publishing updates to their libraries.

After confirming the service was safe to restart with the patched tool, Cloudflare leadership made the decision to not simply ship a stopgap fix. Instead, they reviewed the entire current solution to identify a fundamentally better architecture that would improve the overall security posture of cdnjs.

Forensic Investigation: Confirming Only the Researcher Exploited It

Cloudflare's security team reviewed extensive logging from the auto-update pipeline, which publishes logs to a dedicated GitHub repository, as well as logs retained from cdnjs' Cloudflare account. Based on access logs, API token usage, and file modification metadata, they concluded that only the researcher exploited the vulnerability, and only against test files they published. To rule out further abuse, the team reviewed the list of source IP addresses that accessed the Workers KV token before it was revoked — only one address appeared, belonging to the cdnjs auto-update service itself. A manual review of files pushed to the cdnjs/cdnjs GitHub repository around that time also found no evidence of other abuse.

About half of the libraries on cdnjs update automatically via npm. The attack vector relied on publishing a crafted .tar.gz archive containing a symbolic link to npm. When the cdnjs auto-update pipeline extracted the tarball, it followed the symlink and wrote files outside the intended directory using the pipeline's user privileges.

Cloudflare identified two fundamental design flaws that made this possible:

  • Path traversal: The code extracting untrusted archive files did not adequately verify that output paths remained within the designated folder for the package being processed.
  • Overly broad privileges: The extraction process ran with more permission than necessary for the simple task of unpacking an archive.

Cloudflare patched the path traversal vulnerability by adding an explicit check for each file in the archive. The fix verifies that a file's full canonical path begins with the fully resolved path to the intended destination directory. If the check fails, the system logs a warning and skips extraction of that file. A code comment in the fix notes a residual limitation: if the archive encodes filenames in UTF-8, the path may not be canonicalized properly, and the code may fail to catch a traversal.

Defense in Depth: AppArmor and Full Sandboxing

The path traversal patch was not the final remediation step. Cloudflare also deployed an AppArmor profile for the cdnjs publication pipeline to restrict the service's Linux capabilities at the kernel level, preventing it from performing actions beyond a very narrow set of permitted operations.

/path/to/bin {
  network,
  signal,
  /path/to/child ix,
  /tmp/ r,
  /tmp/cache** rw,
  ...
}

In a sample configuration, the profile constrains the binary to access all networking and signals, execute only specific child processes (which inherit the profile), read from /tmp, and read/write only under /tmp/cache. Every other system access is denied at the kernel layer, limiting the utility of any successful compromise.

With both the path traversal fix and AppArmor in place, Cloudflare deemed this specific attack vector closed. However, having implemented these two layers of defense, they decided that defense in depth alone was insufficient. The team moved forward with a complete redesign of the pipeline to isolate every step and each library, mitigating the entire class of such vulnerabilities.

Redesign: Breaking a Monolith into Microservices

The new cdnjs auto-update architecture departs from a single monolithic process in favor of a set of microservices and daemons, each with a narrow scope of privileges and responsibilities.

BLOG-578 Embedded Image - xXWyQQ

The new process flow operates as follows:

  1. Detection: Two daemons, one for npm-based and one for git-based updates, routinely check for new library versions. When a new version is detected, the files are downloaded as a tarball and placed into an "incoming" storage bucket.
  2. Preparation: A new file in the incoming bucket triggers a function that assembles all metadata required for the update. This function generates a signed URL that permits writing to the "outgoing" bucket, but only within a folder dedicated to that specific library, reducing the impact of any compromised function. The function then adds a message to a queue indicating a publication is ready.
  3. Isolated processing: A daemon watches the queue and, for each message, spawns an unprivileged Docker container to perform the dangerous operations — archive extraction, minification, and compression. Everything happens in this ephemeral, contained environment. After the sandbox exits, the daemon uses the signed URL to write the processed files to the outgoing bucket.
  4. Delivery: Writes to the outgoing bucket trigger several daemons that publish the assets to cdnjs.cloudflare.com and the official cdnjs/cdnjs GitHub repository. These daemons also update the Workers KV store, cdnjs.com, and the cdnjs.com/api endpoint with the finalized package hash and version-specific information.

In this revised design, a vulnerability exploited through a crafted archive would execute within the context of the minimal Docker sandbox. The attacker's access would be limited to ephemeral, library-specific container files; no secrets are available inside, and there is no path to physical host resources or other libraries' data.

Continuous Security Improvement

Cloudflare's response to the cdnjs RCE showcases its commitment to security disclosure. The company maintains a vulnerability disclosure program via HackerOne and regularly contracts third-party security audits. This individual case also displays a willingness to engage deeply with reporting researchers — Cloudflare worked with RyotaK during the disclosure process to ensure the final public post was technically accurate and detailed. Researchers who identify security issues in Cloudflare services are encouraged to submit them through the HackerOne platform.