AI crawlers: three distinct visitors, one discovery pipeline
The web is becoming a two-audience medium: humans and the automated systems trained to serve them. AI bot traffic is rising quickly, and our live tracking shows these crawlers operate in distinct stages that, together, form a discovery funnel. Teams that block all of them indiscriminately risk cutting off a channel before it matures, much like sites that once blocked search engine crawlers and later wondered why their organic traffic vanished.
Not all bots are harmful, and treating them as a monolith is a mistake. Search engine crawlers have been a fact of life for decades, and publishers have long optimized for them. Large language models (LLMs) need training data, and the AI tools built on them need timely, relevant updates to answer user queries. Understanding which bot is visiting, and why, is the difference between turning AI into a growth channel and missing it entirely.
The three stages of AI traffic
AI bot traffic is not a single category. It refers to automated crawlers built to train, update, or run AI systems, distinct from spammy scrapers. Your content moves through a pipeline: it must be crawled to enter training data, referenced from that data to be cited, and cited to generate referrals. A block at any stage removes your content from everything downstream.
Bot traffic already represents a significant share of web activity. Across Vercel deployments last week, bots made up more than 20% of all traffic, with about a quarter of that coming from AI crawlers alone. Some of that is malicious and automatically blocked, but much of it contributes to discovery when handled correctly.
Training crawlers: building the knowledge base
AI training crawlers aim to capture a broad, current view of the web for incorporation into future model responses. Crawlers such as GPTBot and ClaudeBot visit nearly every accessible public page, from documentation and changelogs to product and pricing pages. In our dataset, training crawlers represent the largest share of AI bot traffic because they do not simply revisit popular pages; they strive for full-site coverage.
The value proposition is straightforward: a detailed product description stored in training data lets an AI model later answer a question like "Where can I buy X?" and cite your store as a source. If your pages are never indexed, they cannot surface in those answers.
Grounding bots: fetching current information
Grounding crawlers operate at query time. When someone asks ChatGPT "What's new in Next.js 15?" or asks Perplexity about startups in a given sector, the system checks both its training data and live websites for updates. Content already in the training set can be referenced; content that is not indexed has no chance of being cited.
The upside of being well-indexed is significant. Even a single page can generate hundreds or thousands of mentions across different queries, as observed in real-world usage. A blog post announcing a new feature, for instance, can be picked up by grounding bots within days of publication, letting the AI recommend your product to users searching for related tools.
Referrals: high-intent clicks from AI answers
AI referrals are human visitors arriving from AI-generated responses. These users have typically asked a highly relevant question and received a tailored answer citing your content, so they arrive ready to act. In our network data, referral volume is still lower than traditional search referrals but continues to grow monthly. Some sites report higher conversion rates from this cohort than from organic search visitors.
A concrete case: an AI platform that lists your product among "the best platforms for serverless deployment" will send click-throughs from users who arrive pre-qualified, often ready to evaluate or purchase.
The cost of a blanket block
The historical precedent is instructive. Websites that once blocked Google's crawlers, reasoning that bots only wasted bandwidth, missed the search boom entirely. Developers today risk repeating that mistake with AI crawlers.
AI-powered search already handles billions of queries, and users are discovering content through these platforms alongside traditional search. Blocking AI crawlers severs a growing discovery channel. Unlike traditional SEO, which competes for one of ten blue links, AI systems can draw on your content across countless user queries, recommendations, and comparisons.
Selective access: when blocking makes sense
There are legitimate reasons to restrict AI crawlers. If your content is your product, such as a news site, educational platform, or premium research hub, unlimited AI access could undermine your business model. Beyond that, not every page serves the same purpose, and crawler access should reflect that.
Consider blocking AI crawlers from sensitive routes like /login, /checkout, /admin, and user dashboards. These pages provide no training value and do little for discoverability. At the same time, allow crawlers on discovery content such as documentation, blog posts, landing pages, product pages, and pricing pages, areas where being cited or recommended drives traffic and brand visibility.
The goal is to protect pages that neither explain your product nor answer likely user queries, while keeping high-value discovery content open. Infrastructure like Vercel Firewall, Bot Protection, and BotID can help verify legitimate crawlers, block impersonators, and filter suspicious traffic without shutting down AI-driven discovery.
The shifting baseline for discovery
The web now serves both human visitors and AI systems. As more users turn to AI tools for answers, those tools depend on crawlers to find and recommend content. Bots have always been part of the internet; what is changing is where people seek answers and how those answers are sourced. Sites that make their content accessible position themselves for AI referrals, citations, and authority in their domains. Those that don't risk being overlooked entirely.
