When the customer isn't human
Fewer than half of all HTML page requests now come from a human. A growing share of the rest are AI agents acting on behalf of people: answer engines summarizing options, shopping assistants comparing products, research tools gathering sources. The decision about whether a customer chooses your business increasingly happens inside a model's response, before a human ever loads your homepage.
That shift breaks the old playbook. Discoverability used to mean ranking on a results page. Now it means being found, read, and confidently recommended by machines. Site owners staring at access logs full of AI bots face two questions: can agents actually use my site, and am I getting recommended?
A technical checkup for agent visitors
Agents don't read a site the way people do. A human loads the homepage; an agent leans on robots.txt, sitemaps, response headers, clean Markdown versions of content, and published metadata for authentication and tool discovery. The Agent Readiness Diagnostics scan runs those checks against a hostname and rolls them into a single readiness view, from "Not Ready" to fully agent-native.
Each check comes back as pass, fail, or neutral, with an explanation and an evidence trail showing the exact request and response observed. The checks are grouped by implementation effort:
- Quick wins: crawler-readable
robots.txt, XML sitemap, AI-crawler rules, and serving clean Markdown to agents - Technical groundwork: Content Signals for usage permissions, API catalog, link headers, agent login instructions
- Advanced integration: OAuth discovery, MCP and A2A agent cards, skills index, Web Bot Auth, WebMCP
- Commerce: Emerging agent-payment standards including x402, ACP, UCP, and AP2 — informational only, not counted in the score
Every failed check pairs with a next step. Where Cloudflare has a relevant feature, a "Set up in Cloudflare" link jumps straight to the setting; otherwise a "Copy Agent Prompt" button suggests what your coding agent should build. Make the change, re-scan, and the checkmark turns green.
Measuring AI recommendations
Diagnostics answers whether agents can read a site. The Answer Engine Optimization (AEO) tool answers what happens next: when a customer asks an assistant a question in your category, does it recommend you or a competitor? Unlike a search ranking, you can't look this up — there are no impressions or missed-click reports. When a competitor gets named instead of you, the sale is gone and nothing tells you it happened.
AEO infers your industry and category from your site, then probes leading assistants — currently Anthropic's Claude and OpenAI's GPT — with realistic customer prompts. Models are queried multiple times across different variants via Cloudflare AI Gateway, since assistants rarely answer the same question identically twice. From the responses a customer would actually see, the tool extracts several metrics:
- Citation Rate: the share of category answers that cite your site as a source
- Prominence: how early and how substantially your content appears when cited
- Mention Rate: how often assistants name your brand, regardless of whether your site is cited — read alongside Citation Rate, it separates awareness from attribution
- Share of Voice: your citation share against competitors, showing who is winning the prompts you lose
To score an individual site, AEO builds a panel baseline for each industry and category by querying assistants with prompts that deliberately don't name your brand. That panel is pre-computed once per category and reused across accounts in that domain. That approach gives instant results from a snapshot, avoids redundant AI calls across thousands of scans, and supports an Industry Fit score measuring whether assistants view your site alongside your actual competitors.
Where judgment is required, Workers AI reads each reply and scores citations and mentions natively. The tool also uses exact text analysis rather than letting a model grade its own output, folding dozens of one-off replies into actionable metrics without requiring you to build an evaluation framework.
Alongside the answers, AI Operator Activity shows the real crawl and referral traffic on your site per operator — OpenAI, Google, and others — with the errors they hit along the way, like 403 blocks and 404 dead links. The pattern worth acting on is the operator that crawls thousands of pages but refers no one: it's using your work without sending customers back. Since these numbers are specific to your site, you can experiment, re-run scans, and measure the impact on the exact questions that bring you business.
Most sites aren't built for this audience yet, which is an opportunity. Just as early SEO rewarded sites built for search engines, the sites built for agents will be rewarded now — the ones that are easy to find, read, and trust.
