Cloudflare has extended its AI Crawl Control dashboard beyond reporting: customers can now see which AI services honor their robots.txt rules, which do not, and enforce those rules through the WAF. The feature is live today for all Cloudflare customers.

What robots.txt controls

Robots.txt is a plain text file hosted on a domain that implements the Robots Exclusion Protocol, a standard in use since 1994. It tells crawlers such as Google and Bing which parts of a site they may access. Site owners restrict access for various reasons: keeping content out of search engines or social networks, trusting one platform over another, or reducing automated traffic to their servers.

Generative AI has added a new motive. AI services crawl the Internet to gather training data for proprietary, commercial models that generate new content, so many publishers now use robots.txt to declare policies covering AI bots alongside traditional search engines. As one real-world example from a top online news site shows, a policy can name specific crawlers to block outright:

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

In this case, the site disallows ChatGPT, Anthropic AI, Google Gemini, and ByteDance's Bytespider across all of its content.

From voluntary to enforced

Compliance with the Robots Exclusion Protocol has historically been voluntary. AI Crawl Control now translates the AI bot rules in a site's robots.txt into a WAF rule, so the policy is upheld at the network level rather than merely requested.

How the dashboard works

AI Crawl Control pulls the robots.txt files for a web property, parses them, and matches their rules against the AI bot traffic observed for that property. The summary table aggregates request and violation counts per bot across all paths; hovering over the Robots.txt column reveals the defined policy for each bot in a tooltip. Violations can also be filtered from the top of the page.

BLOG-2619 2

The "Most popular paths" section flags any path receiving traffic that violates the policy. Violations appearing in the Robots.txt column mean a crawler is not complying.

BLOG-2619 3

Deploying enforcement

Clicking "Enforce robots.txt rules" at the top of the summary table converts the rules defined for AI bots into an advanced firewall rule, redirects to the WAF configuration screen, and lets the operator deploy it across Cloudflare's network. The example news site policy becomes the following rule after translation:

BLOG-2619 4

Once such a rule is deployed, the robots.txt policy is enforced rather than requested.

To audit AI bot traffic and enforce robots.txt directives, customers can log into the dashboard and navigate to their domain.