How Cloudflare hunted down its own Log4j exposure

When the Log4j vulnerability (CVE-2021-44228) went public on December 9, 2021, Cloudflare's security team split its response into two tracks: protecting customers and auditing its own environment. The internal track, detailed below, shows how the company identified every Java service running the vulnerable library, patched what it could, and validated that no compromise had occurred.

Assembling the internal picture

The difficult part of responding to a Log4j-class vulnerability is not the fix—it is knowing where the vulnerable software runs. Log4j is embedded in countless Java applications, often as a transitive dependency that developers never explicitly touch. Cloudflare's first step was to build an inventory of every JVM-based service in its centralized code repositories, then check each application for Log4j and determine which version was compiled in.

That scan surfaced three affected services: ElasticSearch, LogStash, and Bitbucket, all running Log4j versions between 2.0 and 2.14.1. While waiting for updated packages, Cloudflare applied the mitigations from the official Log4j security documentation. For each instance, engineers either removed the JndiLookup class from the classpath:

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

Or set the system property log4j2.formatMsgNoLookups in the Log4j configuration.

Parsing the external reports

Even before the internal inventory was complete, Cloudflare was reviewing reports from its HackerOne bug bounty program and a public GitHub post claiming the company was vulnerable. At least two reports suggested Cloudflare had been compromised, including one targeting the developer documentation site at https://developer.cloudflare.com with a proof-of-concept showing a DNS callback.

That report, however, pointed to the IP address 173.194.95.134, which belongs to Google's 173.194.94.0/23 subnet. Cloudflare's developer docs are served by a Cloudflare Worker that only hosts static assets and relies on Google's analytics library. The hypothesis: the attacker received the callback through Google's servers, not Cloudflare's. A similar report submitted through the vulnerability disclosure program could not be reproduced by the researcher, reinforcing that conclusion.

Even if the payload had reached Cloudflare's backend, exploitation would have been blocked. Workers logging is sent to internal services, and Kubernetes egress policies restrict outbound connections to a curated set of internal endpoints—no calls to the public Internet are permitted.

Scanning for active exploitation

While remediation was underway, a separate workstream began checking whether any vulnerable instance had actually been exploited. Cloudflare followed standard incident response practice across four fronts:

Log analysis

The exploit for CVE-2021-44228 relies on recognizable patterns in log messages, such as \$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+. Engineers scanned logs from every potentially impacted service for those patterns.

Network analytics

Cloudflare's network data was reviewed for suspicious inbound and outbound connections, with attention to:

  • Unusual traffic that might indicate active compromise
  • Targeted systems and services, which were correlated against the asset inventory to determine whether any affected host was exposed or exploited
  • Network indicators tied to threat actors attempting to exploit the vulnerability, with outbound traffic to those indicators blocked in Cloudflare Gateway

Endpoint analysis

Findings from log and network reviews informed endpoint scanning criteria. Cloudflare deployed custom Yara detection rules to its Endpoint Detection and Response agent and its centralized SIEM tool. It also analyzed process events across its infrastructure, searching for post-exploitation behavior such as second-stage downloads and anomalous child processes.

Defense in depth pays off

Cloudflare's layered security controls are the reason it believes no compromise occurred. Three specific measures stood out during the investigation:

  1. Restricted outbound traffic. Kubernetes network policies block egress to the Internet on deployments, so even if a service executed the exploit, the next-stage connection to an attacker-controlled server would be dropped.
  2. Authenticated origin pulls. All externally facing services are fronted by Cloudflare, with origin servers never directly exposed to the Internet.
  3. Zero-trust access. Internal services sit behind Cloudflare Access, meaning post-patch exploit attempts would have required authentication.

The company's own WAF also benefited from the customer-facing mitigation effort. New rules written to block Log4j exploitation were shipped to the Cloudflare Specials ruleset with a default action of BLOCK, so Cloudflare received the same protection as its customers without additional configuration.

Third-party providers under review

Beyond its own code, Cloudflare compiled a list of all third-party providers and sub-processors and contacted each one about their exposure to CVE-2021-44228. Responses were still being reviewed at the time of publication. Any critical provider found to be impacted would be disabled and blocked until fully remediated.

A timeline of the response

  • 2021-12-09 16:57 UTC — Hackerone report received regarding Log4j RCE on developers.cloudflare.com
  • 2021-12-10 09:56 UTC — First WAF rule shipped to Cloudflare Specials ruleset
  • 2021-12-10 10:00 UTC — Formal engineering INCIDENT is opened and work begins to identify areas we need to patch Log4j
  • 2021-12-10 10:33 UTC — Logstash deployed with patch to mitigate vulnerability.
  • 2021-12-10 10:44 UTC — Second WAF rule is live as part of Cloudflare managed rules
  • 2021-12-10 10:50 UTC — ElasticSearch restart begins with patch to mitigate vulnerability
  • 2021-12-10 11:05 UTC — ElasticSearch restart concludes and is no longer vulnerable
  • 2021-12-10 11:45 UTC — Bitbucket is patched and no longer vulnerable
  • 2021-12-10 21:22 UTC — Hackerone report closed as Informative after it was unable to be reproduced

At the time of the post, Cloudflare's internal investigation was still ongoing, but across log review, network analytics, and endpoint scanning, the company had found no evidence of compromise.