A dedicated AI Insights section
Cloudflare Radar has added an AI Insights page that consolidates AI-related measurements in one place. It builds on the AI bot & crawler traffic graph first published for Birthday Week 2024 and extends it with views into Generative AI service popularity, robots.txt directives applied to AI bots, and model and task usage on Workers AI.
Bot and crawler traffic
The AI bot & crawler traffic graph, originally launched in September 2024 on Radar's Traffic page, now lives on the AI Insights page. It shows global traffic trends over the selected period for the five most active AI bots and crawlers. The tracked user agent list derives from the ai.robots.txt list and is expanded as new entries are identified. Time series and summary data are exposed through the Radar API, and the full set of observed bots can be browsed in the Data Explorer.

Ranking Generative AI services via 1.1.1.1
The Year in Review reports for 2023 and 2024 already ranked publicly available Generative AI services using request data from the 1.1.1.1 DNS resolver, with ChatGPT on top in both years. Those rankings are retrospective; the new Generative AI services popularity graph tracks relative placement continuously, aggregating DNS request traffic for the associated domains at a daily level. The underlying series can be pulled from the Radar API with serviceCategory=Generative%20AI.
At the end of January 2025, the top five positions had been stable for roughly four weeks while ranks #6–10 churned. DeepSeek entered at #9 on January 26 and reached #3 by January 29.

What robots.txt says about AI access
Sites restrict crawler access through Allow and Disallow directives, but enforcement depends on bots honoring them. Radar now samples its top 10,000 domains weekly, identifies which sites publish robots.txt files, and aggregates the AI-specific directives inside them. The AI user agents found in robots.txt graph classifies each directive as fully or partially allowing or disallowing a named user agent, giving a picture of what high-traffic sites do in practice. Cloudflare's AI Audit offers the complementary owner-side view of how AI bots interact with a given site.

The same data is broken out by domain category, such as Ecommerce or News & Media, to show which bots appear in each category's directives. News & Media sites, for example, almost universally fully disallow AI user agents.

Switching to Allow reveals a far smaller set of user agents and a much smaller group of sites granting full or partial access. Absent an explicit entry and a wildcard *, access is fully allowed by default.

The robots.txt data is also available through the Radar API and the Data Explorer.
Model and task popularity on Workers AI
Radar now reports which publicly available models supported by Workers AI are most used, and which task types those models perform, measured by customer account share. Trends and summary shares for the complete model and task lists are available in the Data Explorer, which also supports comparing shares across time periods. Corresponding model and task popularity endpoints exist in the API.

Keeping pace with a volatile space
New platforms, services and models appear regularly, and entrants can reshape the market quickly, as DeepSeek's rise illustrated. More than two years after ChatGPT's announcement, friction between content providers and AI platforms over scraping for training data persists. The AI Insights page is intended to give observers current data on both fronts as they develop.



