AI crawlers are reshaping how CDN caches must work
Automated traffic now accounts for 32% of Cloudflare's network traffic, according to the company's own data. That figure includes search engine crawlers, uptime checkers, ad networks, and increasingly, AI assistants that pull from the web to power retrieval-augmented generation (RAG) and other real-time services. Unlike human visitors, these agents often issue high-volume, parallel requests, and they tend to scan entire sites sequentially rather than focusing on popular pages. An AI assistant generating a response may fetch images, documentation, and knowledge articles from dozens of unrelated sources in a single session.
Not all operators want to block this traffic. Developers may want their documentation to remain current in foundation models, e-commerce sites may want product descriptions included in LLM search results, and publishers may be looking to monetize access through mechanisms like pay-per-crawl. This leaves website operators with a difficult tradeoff: tune infrastructure for AI crawlers or for human visitors, since the two workloads place very different demands on cache architecture.
Cloudflare's work on this problem is a collaboration with researchers at ETH Zurich. The full findings appeared at the 2025 Symposium on Cloud Computing in the paper "Rethinking Web Cache Design for the AI Era" by Zhang et al.
Caching basics
When a user requests content, the request goes to the Cloudflare data center closest to them. If a valid cached copy exists there, it's served immediately. On a cache miss, the data center reaches back to the origin server for a fresh copy, which then remains in cache until it expires or is evicted. Keeping the right content cached is essential to minimizing misses and delivering fast responses — but what's right for human traffic may be very different from what AI crawlers need.
What makes AI crawler traffic different
AI crawlers are now the most active category of AI bot traffic, accounting for 80% of the self-identified AI bot activity Cloudflare sees. While some crawlers support real-time AI services like summarization and question answering, the vast majority of single-purpose AI bot traffic is devoted to training data collection, with search a distant second.
AI crawler traffic stands apart from other automated workloads in three distinct ways: high unique URL ratio, content diversity, and crawling inefficiency. Public statistics from Common Crawl show that over 90% of pages are unique by content. Different crawlers also target different content types — some focus on technical documentation, others on source code, media, or blog posts. And AI crawlers don't always follow optimal crawling paths; a substantial fraction of their fetches result in 404 errors or redirects, often due to poor URL handling. In addition, AI crawlers typically don't employ browser-side caching or session management the way human users do. Multiple independent instances may each appear as a new visitor to the CDN, even when they request the same content.
Even a single AI crawler tends to dig deeper into websites than a typical human user, exploring a broader range of content. Usage data from Wikipedia shows that pages once considered "long-tail" or rarely accessed are now frequently requested, shifting the distribution of content popularity within a CDN's cache. AI agents may also iterate in loops to refine search results, scraping the same content repeatedly — a pattern that leads to low content reuse and broad coverage.
As AI agents loop iteratively to refine search results — a common RAG pattern — they maintain a consistently high unique access ratio, typically between 70% and 100%. Each loop increases accuracy for the agent but constantly fetches new, unique content rather than revisiting previously seen pages. This repeat access to long-tail assets churns the cache that human traffic relies on, which could make existing pre-fetching and traditional cache invalidation strategies less effective as crawler traffic grows.
Impact on cache hit rates
For a CDN, a cache miss means a trip to the origin server, which slows response times and increases egress costs and origin load. AI crawlers' broad, unpredictable access patterns with long-tail reuse significantly raise the cache miss rate, and standard techniques for improving cache hit rates — such as cache speculation or prefetching — become noticeably less effective.
Cloudflare measured the difference in cache hit rates for a single CDN node with and without identified AI crawlers. While the impact remains limited today, there's a clear drop in hit rate when AI crawler traffic is included. The cache is managed with a least recently used (LRU) algorithm, meaning the least-requested content is evicted first when storage is full. The drop in hit rate suggests LRU struggles under the repeated scan behavior of AI crawlers.
Real-world effects are already visible. Wikimedia experienced a 50% surge in multimedia bandwidth usage due to bulk image scraping. Fedora, which hosts large software packages, and the Diaspora social network suffered from heavy load and poor performance for human users. Many other sites have reported bandwidth increases or slowdowns from AI bots repeatedly downloading large files. Blocking crawler traffic mitigates some impact, but a smarter cache architecture would allow site operators to serve AI crawlers without degrading response times for humans.
AI-aware cache architecture
AI crawlers power latency-sensitive applications like RAG and real-time summarization, so those requests should be routed to caches that balance larger capacity with moderate response times. These caches must preserve freshness but can tolerate slightly higher access latency than human-facing caches.
AI crawlers also support training-set construction and large-scale content collection, which can tolerate significantly higher latency and aren't time-sensitive. Requests for these workloads could be served from deep cache tiers that take longer to reach, such as origin-side SSD caches, or delayed with queue-based admission control and rate-limiters to prevent backend overload. This also creates an opportunity to defer bulk scraping when infrastructure is under load, without affecting interactive use cases.
Existing tools like Cloudflare's AI Index and Markdown for Agents already let website operators present simplified versions of their sites to known AI agents. The company is planning further work to mitigate AI traffic's impact on CDN cache. Two complementary approaches are under exploration with ETH Zurich collaborators: traffic filtering with AI-aware caching algorithms, and the addition of an entirely new cache layer dedicated to AI crawler traffic.
Cache replacement algorithms determine which content gets evicted when a cache is full. LRU, LFU (least frequently used), and FIFO (first-in, first-out) are the common options, with LRU generally considered the best balance of simplicity, low overhead, and effectiveness for generic situations. For mixed human and AI bot traffic, however, initial experiments indicate that other algorithms, particularly SIEVE or S3FIFO, could allow human traffic to achieve the same hit rate with or without AI interference. Cloudflare is also experimenting with workload-aware, machine learning-based caching algorithms that could customize cache response in real time.
Long term, the expectation is that a separate cache layer for AI traffic will be the best approach. Human traffic would continue to be served from edge caches at CDN points of presence, which prioritize responsiveness and cache hit rates. AI traffic would be handled differently depending on task type.
A growing problem
The impact of AI bot traffic on cloud infrastructure will only intensify in the coming years. Better characterization of AI traffic effects on CDNs across the globe is needed, along with new cache policies and architectures to address this novel workload. Cloudflare is already working to reduce bandwidth costs for customers with high bot traffic through AI-aware caching, and its AI Crawl Control and Pay Per Crawl tools offer customers more control over programmatic access to their content.
