Migration misfire reroutes 24 hostnames to ClickFunnels
On January 24, 2022, Cloudflare mistakenly rerouted traffic for 24 hostnames to the ClickFunnels origin during an internal product migration. The error began at 22:24 UTC when Cloudflare started moving hundreds of thousands of custom hostnames from SaaS v1 (the legacy system) to SaaS v2, a migration intended to be seamless and transparent.
The migration process read custom hostname configurations from a database and transferred them automatically. Custom hostnames, which allow SaaS providers to manage their customers' websites and SSL certificates at scale, normally require DNS validation before they become active and can be proxied. Once validated, a hostname moves from pending to active status, except when it is blocked within Cloudflare's system. Blocked hostnames require manual approval from the Trust & Safety team and include Cloudflare-owned properties and well-known brands.
During the migration of clickfunnels.com's customers, a small number of blocked hostnames were incorrectly moved to the active state. This caused traffic for those hostnames to be processed by clickfunnels.com's configuration and sent to their origin. ClickFunnels was unaware of the issue until traffic began hitting their servers.
Impact timeline and scope
The affected hostnames included www.cloudflare.com, but not the apex domain cloudflare.com. Subdomains such as dash.cloudflare.com, api.cloudflare.com, and cdnjs.cloudflare.com were unaffected. Visitors to impacted sites may have seen a clickfunnels.com page instead of the intended content, and API calls to affected hostnames may have failed or returned unexpected responses.
The timeline unfolded quickly:
- 23:06 UTC — www.cloudflare.com traffic was rerouted.
- 23:15 UTC — an internal incident was declared.
- 23:34 UTC — www.cloudflare.com was restored, root cause identified, and remediation for remaining hostnames began.
- 00:13 UTC (January 25) — all affected hostnames were restored and the incident closed.
Of the 268,430,157 total requests redirected during the event, 268,220,296 (99.92%) were for www.cloudflare.com.
Follow-up actions
Cloudflare has contacted all affected customers and worked with ClickFunnels to delete logs of the event, ensuring no erroneously sent data is retained. To prevent recurrence, Cloudflare plans several changes:
- Blocked hostname overrides will no longer be permitted; all changes will route through the verification pipeline during migration.
- SaaS customers must provide explicit validation and approval before a blocked hostname can be activated.
- Additional monitoring will be added to hostnames being migrated to detect erroneous traffic patterns.
- Additional monitoring will be added for www.cloudflare.com.
- Hostname activations will be staged on non-production elements first to verify the expected state before promoting to production.
Cloudflare acknowledged that the incident exposed previously unknown gaps in its process and technology, and apologized for the disruption to customers and visitors of the impacted properties.
