AI crawlers: a growing share of web traffic with real limitations

AI crawlers have become a substantial presence on the web. Data from Vercel's network and MERJ's analysis shows OpenAI's GPTBot generated 569 million requests in the past month, while Anthropic's Claude followed with 370 million. Combined, that's roughly 20% of Googlebot's 4.5 billion requests during the same period.

The study monitored nextjs.org and the Vercel network, then validated findings against two job board sites built on different stacks: Resume Library (Next.js) and CV Library (custom monolithic framework). Microsoft Copilot was excluded because it lacks a unique user agent for tracking.

Traffic volume and distribution

Across Vercel's network in the past month, crawler fetch volumes broke down as follows:

  • Googlebot: 4.5 billion fetches across Gemini and Search
  • GPTBot (ChatGPT): 569 million fetches
  • Claude: 370 million fetches
  • AppleBot: 314 million fetches
  • PerplexityBot: 24.4 million fetches

GPTBot, Claude, AppleBot, and PerplexityBot together account for nearly 1.3 billion fetches—a little over 28% of Googlebot's volume. While they haven't reached Google's scale, they represent a meaningful share of crawler traffic.

All measured AI crawlers operate from U.S. data centers: ChatGPT from Des Moines and Phoenix, Claude from Columbus. Traditional search engines spread crawling across more regions—Googlebot, for instance, operates from seven U.S. locations including The Dalles, Council Bluffs, and Moncks Corner.

JavaScript: fetching vs. executing

None of the major AI crawlers currently render JavaScript. This includes OpenAI's crawlers (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic's ClaudeBot, Meta's Meta-ExternalAgent, ByteDance's Bytespider, and PerplexityBot.

Two exceptions stand out. Google's Gemini leverages Googlebot's infrastructure, which provides full JavaScript rendering. AppleBot runs a browser-based crawler that processes JavaScript, CSS, and Ajax requests similar to Googlebot. Common Crawl's CCBot, often used as a training dataset for LLMs, does not render pages either.

The data shows ChatGPT and Claude do fetch JavaScript files—11.50% and 23.84% of requests respectively—but they don't execute them, so client-side rendered content remains unreadable. Content in the initial HTML response, such as JSON data or delayed React Server Components, may still be indexed since AI models can interpret non-HTML content.

Content type preferences

AI crawlers show distinct content priorities on nextjs.org:

  • ChatGPT prioritizes HTML content (57.70% of fetches)
  • Claude focuses heavily on images (35.17% of total fetches)
  • Both spend significant time on JavaScript files despite not executing them

Googlebot's fetches are more evenly distributed: 31.00% HTML, 29.34% JSON, 20.77% plain text, and 15.25% JavaScript. The pattern suggests AI crawlers collect diverse content types—HTML, images, and JavaScript as text—likely for training purposes. Traditional search engines have optimized crawling for indexing, while newer AI companies may still be refining their prioritization strategies.

Crawling inefficiencies

The data reveals notable inefficiencies in AI crawler behavior:

  • ChatGPT spends 34.82% of fetches on 404 pages
  • Claude hits 404s on 34.16% of fetches
  • ChatGPT spends an additional 14.36% of fetches following redirects

Excluding robots.txt, the 404 errors frequently involve outdated assets from the /static/ folder, suggesting a need for better URL selection. Googlebot, by contrast, spends only 8.22% of fetches on 404s and 1.49% on redirects, indicating more refined targeting of real resources.

Traffic pattern analysis from nextjs.org shows pages with higher organic traffic receive more frequent crawler visits, but AI crawlers exhibit less predictable URL selection overall. Their high 404 rates suggest validation processes still need work.

Recommendations for site owners

If you want to be crawled

  • Server-render critical content. Since ChatGPT and Claude don't execute JavaScript, main content, meta information, and navigation should be server-rendered. SSR, ISR, and SSG keep content accessible to all crawlers.
  • Client-side rendering is fine for enhancements. Non-essential dynamic elements like view counters, interactive UI features, live chat, and social feeds can safely use client-side rendering.
  • Manage URLs carefully. High 404 rates highlight the need for proper redirects, up-to-date sitemaps, and consistent URL patterns.

If you don't want to be crawled

  • Use robots.txt. It works for all measured crawlers. Set specific rules for AI crawlers using their user agent or product token (check each company's documentation for exact identifiers).
  • Block at the network level. Vercel's firewall rule can deny AI crawler access with one click.

For AI users, several caveats apply. Responses about dynamic web applications may be incomplete since ChatGPT and Claude can't see JavaScript-rendered content. High 404 rates mean AI-provided URLs may be incorrect—verify sources directly for critical information. Freshness can also be inconsistent; even when ChatGPT or Claude claim to have fetched the latest data from nextjs.org, server logs often show no corresponding fetches, suggesting reliance on cached or training data.

The bottom line

AI crawlers now drive nearly 1 billion monthly requests across Vercel's network, but their behavior differs markedly from traditional search engines in rendering capabilities, content priorities, and efficiency. Standard web development best practices—particularly around content accessibility—remain the best defense as these crawlers evolve.