Thanksgiving 2023 security incident

Cloudflare Details Thanksgiving Intrusion, “Code Red” Remediation

On November 23, 2023, Cloudflare detected a threat actor on its self-hosted Atlassian server. The company’s security team cut off access that day, and on November 26 brought in CrowdStrike’s forensic team for an independent analysis. CrowdStrike completed its investigation this week, and Cloudflare has now published a detailed account of the incident.

Cloudflare states that no customer data or systems were impacted. The company attributes the limited blast radius to its access controls, firewall rules, and hard security keys enforced via its own Zero Trust tools. The threat actor’s lateral movement was constrained, no services were implicated, and no changes were made to global network systems or configuration.

The company believes the attacker was a nation-state actor seeking persistent, widespread access to Cloudflare’s global network. The intrusion was discovered on Thanksgiving Day, and the last evidence of threat activity was on November 24 at 10:44 UTC. The actor used one access token and three service account credentials that had been taken during the Okta compromise of October 2023 and that Cloudflare failed to rotate.

Initial Access and Activity Timeline

The attack began with the October 18 Okta compromise, but the threat actor only started targeting Cloudflare systems in mid-November. The credentials that were not rotated included a Moveworks service token granting remote access to Atlassian, a Smartsheet service account with administrative access to Atlassian Jira, a Bitbucket service account for the source code management system, and an AWS environment with no access to the global network or sensitive data. These credentials were mistakenly believed to be unused.

From November 14 to 17, the threat actor performed reconnaissance. They attempted to log into Cloudflare’s Okta instance and the Cloudflare Dashboard, and were denied both times. They did access an AWS environment used for the Cloudflare Apps marketplace, which was segmented from the global network. That service account was revoked.

On November 15, the threat actor gained access to Atlassian Jira and Confluence using the Moveworks token to authenticate through the gateway and the Smartsheet account for Atlassian suite access. Over the following days, they searched the wiki for terms like remote access, secret, client-secret, openconnect, cloudflared, and token. They accessed 36 Jira tickets (out of 2,059,357 total) and 202 wiki pages (out of 194,100 total), looking for information about the configuration, security, and management of Cloudflare’s global network.

On November 16, the threat actor used the Smartsheet credential to create an Atlassian account that appeared to be a normal Cloudflare user and added it to groups for persistent access. They then took a break until November 20, apart from brief connectivity tests.

Persistence and Lateral Movement Attempts

On November 22, the threat actor used the Smartsheet account’s administrative access to Jira to install the Sliver Adversary Emulation Framework via the ScriptRunner for Jira plugin. This provided command-and-control connectivity and persistent access to the Atlassian server. They attempted to use this access to reach a non-production console server in Cloudflare’s new São Paulo, Brazil data center, but were denied. That data center was not yet in production.

Over the next day, the threat actor viewed 120 code repositories (out of 11,904 total). Using the Atlassian Bitbucket git archive feature, they downloaded 76 repositories to the Atlassian server. Cloudflare could not confirm whether these were exfiltrated, so it treated them as having been. The repositories were almost all related to backups, global network configuration and management, identity, remote access, and usage of Terraform and Kubernetes. A small number contained encrypted secrets, which were rotated immediately.

Discovery and Termination of Access

Cloudflare’s security team was alerted on November 23 at 16:00 UTC. The detailed timeline of the response was as follows:

• 15:58 – Threat actor adds Smartsheet service account to an administrator group.
• 16:00 – Automated alert about the change.
• 16:12 – Cloudflare SOC begins investigating.
• 16:35 – Smartsheet service account deactivated.
• 17:23 – Threat actor-created Atlassian user account found and deactivated.
• 17:43 – Internal incident declared.
• 21:31 – Firewall rules block threat actor’s known IPs.

On November 24, the last threat activity was recorded at 10:44, and Sliver was removed at 11:59. All threat actor access and connections were terminated.

“Code Red” Remediation Effort

On November 27, Cloudflare redeployed a large portion of its technical staff to a project dubbed “Code Red.” The focus was strengthening, validating, and remediating controls, while also ensuring the threat actor had no persistent access. CrowdStrike’s independent assessment corroborated Cloudflare’s findings and did not uncover any missed activity.

The remediation effort included the following actions:

• Rotating every production credential — more than 5,000 individual credentials.
• Physically segmenting test and staging systems.
• Performing forensic triages on 4,893 systems.
• Reimaging and rebooting every machine in the global network, including all systems the threat actor accessed and all Atlassian products.
• Returning equipment from the Brazil data center to manufacturers for forensic examination. No access or persistence was found, but the hardware was replaced anyway.
• Searching for and deleting all HAR files uploaded to the wiki that might contain tokens.
• Reviewing source code repositories for embedded secrets, which were rotated.

The immediate “Code Red” effort ended on January 5, but ongoing work continues around credential management, software hardening, vulnerability management, and additional alerting.

What the Threat Actor Did Not Access

Cloudflare saw no evidence that the threat actor accessed the global network, data centers, SSL keys, customer databases or configurations, Cloudflare Workers, AI models, network infrastructure, or datastores like Workers KV, R2, or Quicksilver. The actor also attempted to access internal metrics, network configuration, build systems, alerting, and release management systems, but none of those attempts were successful. Their access was limited to the Atlassian suite and the server running it.

Cloudflare is publishing the following indications of compromise (IOCs) so that other organizations, particularly those that may have been impacted by the Okta breach, can search their logs to confirm the same threat actor did not access their systems.