Canonical Tags, Enforced as Redirects for AI Crawlers
Cloudflare has launched Redirects for AI Training, a feature that turns a site's existing canonical tags into enforced HTTP redirects specifically for verified AI training crawlers. The feature is available on all paid Cloudflare plans via a single toggle in the dashboard.
The motivation is straightforward: AI training crawlers consume deprecated content at nearly the same rate as current content, rendering traditional advisory signals like deprecation banners and noindex meta tags ineffective. Cloudflare reports that on its own documentation site, bots in the AI Crawler category made 4.8 million visits over 30 days, with no measurable difference in consumption between deprecated and current pages. Because AI agents often rely on trained models rather than live fetches, ingesting outdated content compounds the problem over time.
For context, search engines have a rich noindex signal system, but there is no equivalent HTML directive that says "do not train on this." Blocking AI crawlers entirely creates a void, offering no guidance on what the crawler should learn instead. robots.txt requires exhausting per-crawler, per-path manual upkeep. The <link rel="canonical"> tag—already present on 65-69% of web pages and defined in RFC 6596—already declares which URL is the authoritative version of a page. Redirects for AI Training simply enforces it.
How the Redirect Mechanism Works
The feature uses two inputs: Cloudflare's cf.verified_bot_category field and the <link rel="canonical"> tags in a site's HTML. The AI Crawler category covers bots like GPTBot, ClaudeBot, and Bytespider, which is distinct from the AI Assistant and AI Search categories that cover AI agents.
When a verified AI crawler requests a page with a non-self-referencing canonical tag, Cloudflare issues a 301 Moved Permanently redirect to the canonical URL before the response is returned. Human traffic, search engine indexing, and other automated traffic are unaffected.
Scope Limitations
The feature does not retroactively correct training data already ingested, nor does it cover unverified crawlers outside the AI Crawler category. Cross-origin canonical tags are ignored by design, as they are typically used for domain consolidation rather than content freshness. Self-referencing canonical tags also do not trigger redirects, preventing loops.
Why Not Standard Redirect Rules?
Single Redirect Rules can target AI crawlers via user-agent strings, which works for sites with only a handful of deprecated paths. However, this approach does not scale: every new deprecated path requires updating a rule, user-agents must be manually tracked, and the rules consume plan allocation better used for campaign URLs or domain migrations. Redirect rules also re-encode what canonical tags already declare, and they fall out of sync as content changes.
Evidence from Cloudflare's Own Documentation
Cloudflare's experience illustrates the severity of the problem. In March 2026, legacy Workers documentation on developers.cloudflare.com was crawled approximately 46,000 times by OpenAI, 3,600 times by Anthropic, and 1,700 times by Meta—despite running AI Crawl Control.
That deprecated content ingestion aligns with an observed incident: a leading AI assistant gave an outdated answer in April 2026 about writing KV values via the Wrangler CLI. It recommended kv:key put, but the correct syntax was wrangler kv key put. The colon syntax was deprecated in Wrangler 3.60.0, and while the documentation properly showed a deprecation notice, its influence on training pipelines was unclear.
After enabling Redirects for AI Training on developers.cloudflare.com, Cloudflare found that 100% of AI training crawler requests to pages with non-self-referencing canonical tags were redirected within the first seven days. The crawlers were no longer served deprecated content. Whether that improves AI-generated answers about legacy tools remains an open hypothesis, given how closed training pipelines are, but what crawlers receive at the point of access improved immediately.
Enabling the Feature
For sites with canonical tags already in place, enabling the feature requires no new configuration. Cloudflare's verified bot classification handles crawler identification automatically.
In the dashboard: navigate to AI Crawl Control > Quick Actions > Redirects for AI training and toggle it on for any domain. For path-specific control via Configuration Rules and Cloudflare for SaaS, Cloudflare points to its full documentation.
Cloudflare has also added a response status code analysis view to Radar's AI Insights page. It breaks down the 2xx, 3xx, 4xx, and 5xx status codes that AI crawlers receive across all Cloudflare traffic, providing a visibility layer into how the web currently responds to AI training requests.
Status codes reveal how websites treat AI crawlers
A 301 Moved Permanently is the enforcement mechanism behind Redirects for AI Training, but it is only one response in a wider exchange between origins and crawlers. A 200 OK means content was delivered. A 403 Forbidden blocks access. A 402 Payment Required signals that the client must pay to access content. The spread of these codes across AI crawler traffic shows, at scale, how the web is actually responding.
Cloudflare Radar's AI Insights page now includes a Response status code analysis graph showing the share of top status codes or code groupings (selectable via a dropdown) for AI crawler traffic. Data can be filtered by industry set, and the crawl purpose filter is available in Data Explorer. Those filters help reveal whether crawler types behave differently or whether request patterns vary by industry.
In the general view, just over 70% of requests were served successfully (200), 10.1% were redirected (301, 302), and 3.7% hit missing files (404). Access was blocked for 8.3% of requests via 403. Grouped by category, nearly 74% of requests got successful responses (2xx), 13.7% got client error responses (4xx), 11.3% got redirection messages (3xx), and 1.2% got server error responses (5xx).
Per-bot pages also include this analysis. For GPTBot, the example shows just over 80% of requests served successfully (200), 4.7% redirected (301, 302), and 2.7% for missing files (404). Nearly 6% were blocked with a 403 from Cloudflare. Grouped, 83% of requests received 2xx responses, nearly 10% received 4xx, 5.1% received 3xx, and the remaining 2.2% got 5xx.
Data Explorer supports deeper drill-downs. Users can ask which crawlers request the most non-existent content (yielding 404) and how that traffic trends over time, or which industries send the most Redirection (3xx) codes to Training crawlers.
Status code data is available in aggregate and per bot through the Cloudflare Radar API.
Redirects for AI Training controls what crawlers receive from your origin; Radar's status analysis shows what the rest of the web is doing. Enable it in AI Crawl Control > Overview > Quick Actions to turn advisory signals into enforced outcomes.
Questions or findings? Join the Cloudflare Community or find the team on Discord.
