Configurable Bot Protection for Custom Crawlers
Cloudflare's Super Bot Fight Mode blocks an average of 336 million bot requests per day for self-service customers. While the service’s Verified Bot directory allows trusted automated traffic from major crawlers like Google and Bing, many customers run their own localized scripts for maintenance or content crawling. These custom bots don’t qualify for verification but still need to pass through Super Bot Fight Mode.
Previously, customers had no way to configure exceptions for these single-site bots. The service operated independently on Cloudflare’s global network, isolated from other security tools. That has changed with a behind-the-scenes rewrite that transforms Super Bot Fight Mode into a managed ruleset within the new WAF, alongside the OWASP Core Ruleset and Cloudflare Managed Ruleset. The interface remains the same, but the feature now sits closer to where customers already manage their other security rules.
Skip Rules Replace Allow and Bypass
The WAF team is currently migrating all self-serve customers from the legacy Firewall Rules system to Custom Rules. This transition introduces a single skip action that consolidates the old allow and bypass actions. Rules using skip can bypass remaining custom rules, just like an allow rule, but they can also skip additional Cloudflare services. As customers gain access to the "skip" action, they will see an option to exempt traffic from Super Bot Fight Mode directly in the rule builder.
The example below shows how configurations for exempting traffic from Super Bot Fight Mode will appear:
WordPress Loopback Exemption
One recurring customer use case for skipping Super Bot Fight Mode did not fit the standard pattern: WordPress Loopback requests. As part of its self-diagnostic capabilities, WordPress sites make automated requests back to themselves over the Internet to verify reachability and functionality. These loopback checks come from a wide range of community-developed plugins, each implementing the requests differently. To support this diversity of diagnostic tools, Cloudflare has added a configuration option that securely allows these loopback requests without opening the broader bot protection gap.
Cloudflare plans to integrate this capability into its WordPress plugin in the future, aiming to simplify WordPress configuration alongside bot protection settings.
Rollout and Future Plans
Self-serve customers with Custom Rules can create skip rules to exempt traffic from Super Bot Fight Mode today. The rollout of Custom Rules to all customers is underway; those without the option should see it available within a few weeks.
The rewrite also lays groundwork for Free plan customers. With the migration of Super Bot Fight Mode to the WAF complete, Cloudflare plans to apply the same migration to the original Bot Fight Mode, potentially bringing customizable bot protection to the free tier.
