Announcing Friendly Bots

Cloudflare already maintains a set of manually verified good bots — Google's crawler and Stripe's payment bot among them — which customers can allowlist. The bottleneck is throughput: new bots appear faster than the company can vet them. Friendly Bots is the answer, letting customers self-validate traffic that merits a place between bad and verified.

Four ways to prove a bot is legitimate

A bot operator submits its name, public documentation and user agent (or regex) through Cloudflare's public form. Verification itself rests on one of four mechanisms:

  • IP list. A published list of addresses exclusive to the bot. It may be dynamic: point Cloudflare at a URL and updates are fetched daily. Shared addresses, such as those belonging to a proxy, trip risk detection and are refused.
  • rDNS. The operator supplies a hostname suffix, and reverse DNS maps incoming IPs back to that domain to confirm identity.
  • User agent plus ASN. Traffic that consistently arrives from one network with one user agent can be validated, though this is not always possible because such traffic is easier to spoof.
  • Machine learning. Cloudflare handles 32+ million requests per second and feeds them to a model that profiles good bots. A candidate needs thousands of requests before a usable pattern emerges.
BLOG-952 Embedded Image - PjgPvF

Manual approval typically lands within a few weeks but is largely reserved for trusted partners and large bots, leaving smaller operators without a path.

Where the existing process falls short

The gap shows up in several recurring scenarios: a personal monitoring service crawling a handful of your own sites; a proxy labeled automated because humans and bots both use it, which you want to allow only on your site; automated services at a startup that have not reached global scale. None of these satisfy the criteria for Verified Bots — serving the broader Internet, demonstrating objectively good behavior, and complying with standards such as robots.txt — yet each still needs to get through.

How Friendly Bots works

In the Cloudflare dashboard you describe the bot with an IP list, a hostname suffix, or other methods including machine learning. Each mechanism runs in every Cloudflare data center over the anycast network, so validation is performant, secure and scalable. The resulting collection can be shared across sites and driven by custom policies that allow, rate limit or log the traffic — or simply keep tabs on a bot. Response options integrate with the Workers platform.

BLOG-952 Embedded Image - 4GC37q

The prior system relied on a cache of verified traffic, so a bot had to make thousands of requests to Cloudflare to produce a signal. Friendly Bots drops that requirement with a new dynamic cache built for smaller projects.

What customers gain

  • Verification gets easier. When many customers allowlist a given IP or hostname suffix, the system queues that bot for verification automatically.
  • Validation is near-instant. A broken piece of data — say, an rDNS entry naming the wrong hostname — shows up immediately, because the bot simply is not allowlisted. No waiting weeks on support.
  • Bot owners are no longer a dependency. Previously only the operator could submit verification data for its bot; now anyone who can find identifying information can register a bot on their site.
  • No arbitration. Whether a scraper is desirable is up to you to categorize at the domain level, while Cloudflare keeps verifying bots globally where behavior is objectively good.

Radar and logging get Verified Bot data

Alongside Friendly Bots, Cloudflare Radar now publishes a Verified Bots list, the first detailed listing of its kind. Data updates in real time, with new bots appearing as they are verified.

BLOG-952 Embedded Image - 3A8xNP

Verified bots are also starting to surface in the Logs product as Bot Tags: a request from Pinterest's bot, for example, can carry the string pinterest.

Availability

Friendly Bots is launching soon. Cloudflare expects the effect to ripple through Bot Management — reducing false positives, improving crawlability and generally stabilizing sites. Customers with Bot Management who want in ask their account team; suggestions for bots that should be verified go through the submission form.

BLOG-952 Embedded Image - rxDAEI