Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

Perplexity’s crawler caught dodging blocks and masks

Cloudflare says it has observed Perplexity’s AI crawler using hidden, undeclared user agents to bypass website blocks after its public bots are denied access. The company reports that Perplexity has been rotating IP addresses and switching source ASNs to obscure its identity, while also ignoring or failing to fetch robots.txt files.

The finding stems from customer complaints: sites that had explicitly disallowed Perplexity’s crawlers — both PerplexityBot and Perplexity-User — via robots.txt and WAF rules were still seeing the content accessed. Cloudflare confirmed its blocks were working on the published pages, then set up fresh, unpublished test domains to observe the behavior directly.

Test domains and the hidden crawler

Cloudflare created brand-new domains (e.g., testexample.com and secretexample.com) that were not indexed by search engines or otherwise publicly reachable. Each domain was configured with a robots.txt that disallowed all respectful bots:

Despite these precautions, querying Perplexity’s answer engine about those domains returned detailed information about the content hosted on them.

Obscured identity and rotating infrastructure

When Cloudflare’s test domains blocked Perplexity’s declared crawler user agents, the company observed a second, undeclared crawler using a generic user agent that impersonates Google Chrome on macOS. This stealth crawler accessed content for scraping, contrary to the norms in RFC 9309.

The undeclared crawler used IPs that are not in Perplexity’s official published range and rotated through them in response to blocks. It also switched ASNs in attempts to further evade detection. Cloudflare says it fingerprinted the crawler using machine learning and network signals, across tens of thousands of domains and millions of requests per day.

When the stealth crawler was successfully blocked, Perplexity fell back on other data sources to build answers. Those answers were noticeably less specific and lacked details from the original pages, confirming the blocks were effective.

What well-behaved crawlers should do

Cloudflare contrasts Perplexity’s behavior with its Verified Bots Policy, which outlines expectations for legitimate bots:

• Be transparent. Identify with a unique user agent, publish IP ranges or use web bot authentication, and provide contact info.
• Be well-behaved netizens. Avoid flooding sites, scraping sensitive data, or using stealth tactics to dodge detection.
• Serve a clear purpose. Define the bot’s function publicly so site owners can look it up.
• Separate bots for separate activities. Use distinct bots for distinct tasks so site owners can allow or block selectively.
• Follow the rules. Respect robots.txt, stay within rate limits, and never bypass security protections.

Cloudflare points to OpenAI as a counterexample. OpenAI documents its crawlers and their purposes, respects robots.txt, and does not evade network-level blocks. In the same test setup, ChatGPT-User fetched the robots file and stopped crawling when disallowed, with no follow-up crawls from other user agents. When presented with a block page instead, it also stopped, again with no attempts from other agents.

Mitigations for site owners

Cloudflare states that its bot management system scored all of the observed undeclared crawling as bot traffic, meaning existing block rules on that system already protect customers. Those who prefer not to outright block can set up challenge rules, letting human users through while keeping the crawler out. Additionally, Cloudflare has added signature matches for the stealth crawler to its managed rule that blocks AI crawling activity, available to all customers, including free-tier users.

Outlook

Cloudflare notes that over 2.5 million websites now disallow AI training via its managed robots.txt feature or the managed AI crawler blocking rule. The company acknowledges that crawler evasion techniques will keep evolving — what it observed will almost certainly change — and that its countermeasures will be updated accordingly. It is also working with the IETF on standard extensions to robots.txt to establish clearer, measurable principles for bot operators.