Forget IPs: using cryptography to verify bot and agent traffic

Why IP-based bot verification is falling apart

Bot traffic is no longer a simple binary. Security threats like credential stuffing and denial-of-service attacks sit alongside benign automated visitors — search engine crawlers, RSS fetchers, and AI agents that site owners may actively want to engage with. The traditional identification methods are straining under this new reality.

Cloudflare has historically relied on two signals to verify legitimate crawlers: User-Agent headers and IP address validation. The User-Agent header, defined in RFC 9110, lets bot developers identify themselves, but it is trivially spoofable. IP validation, which checks published address ranges, is more robust but brittle: ranges are frequently shared across users and services — particularly with privacy proxies, VPNs, and shared hosting platforms like Cloudflare Workers — and they shift as infrastructure changes.

The industry needs an affirmative mechanism for legitimate agents to prove who they are. Cloudflare's answer is cryptographic request authentication. Two proposals — HTTP message signatures and request mTLS — give friendly bots a way to sign their traffic and give customer origins a way to verify it.

The limits of current bot identification options

Developers building agents today have three primary options for identifying their HTTP traffic to other services, and none are adequate:

• User agent headers. These are simple to set but easily forged. The situation is worse for semi-automated browsers: agents often masquerade with a Chrome user agent, a practice RFC 9110 explicitly discourages, noting that recipients may assume the masquerading user intentionally desires responses tailored to that identified agent.
• Published IP ranges. The same IP can serve multiple users, services, or even companies when infrastructure is shared. Addresses also change with underlying infrastructure, forcing ad-hoc sharing mechanisms like CIDR lists.
• Pre-shared secrets. Distributing Bearer tokens to every website requires maintaining separate credentials per destination — impractical at web scale.

The proposed alternative is straightforward: bot and agent developers cryptographically sign each outgoing request. Reverse proxies like Cloudflare, protecting origins on behalf of site owners, can then validate those signatures to confirm the request's source.

A typical system involves three actors:

• User: the entity performing actions on the web, whether human, automated program, or anything retrieving information.
• Agent: an orchestrated browser or software program — Chrome on your machine, or OpenAI's Operator with ChatGPT. Agents interact with the web per standards: HTML rendering, JavaScript, subrequests.
• Origin: the website hosting the resource, Cloudflare for proxied sites, or the customer's own exposed servers.

HTTP Message Signatures

HTTP Message Signatures, standardized in RFC 9421, provides cryptographic authentication of a request sender. It's not the only signing framework — AWS uses Signature v4, Stripe authenticates webhooks with its own scheme — but Message Signatures is a published standard and the cleanest developer-facing option.

Industry adoption is underway. OpenAI, for instance, now signs requests from Operator. In the words of Eugenio, an engineer at OpenAI: "Ensuring the authenticity of Operator traffic is paramount. With HTTP Message Signatures (RFC 9421), OpenAI signs all Operator requests so site owners can verify they genuinely originate from Operator and haven't been tampered with."

How signing works

Before sending a request, the agent signs the target origin's authority — for https://example.com/path/to/resource, it signs example.com — with a public key known to the origin. The agent then adds a Signature-Input header containing:

1. A validity window (created and expires timestamps)
2. A Key ID uniquely identifying the signing key, expressed as a JSON Web Key Thumbprint
3. A tag indicating the signature's purpose and validation method, such as web-bot-auth for bot authentication

A companion Signature-Agent header tells the origin where to find the public keys used for signing — for example, a directory hosted at crawler.search.google.com for Google Search, operator.openai.com for OpenAI Operator, or a workers.dev domain for Cloudflare Workers. This header is itself part of the signed content.

GET /path/to/resource HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 Chrome/113.0.0 MyBotCrawler/1.1
Signature-Agent: signer.example.com
Signature-Input: sig=("@authority" "signature-agent");\
             	 created=1700000000;\
             	 expires=1700011111;\
             	 keyid="ba3e64==";\
             	 tag="web-bot-auth"
Signature: sig=abc==

In the example above, the User-Agent lists Chrome first because the agent uses an orchestrated Chrome browser; MyBotCrawler/1.1 follows in decreasing order of importance, per HTTP conventions.

Signature overhead at internet scale is a real concern, but with an appropriate cryptographic suite, the computational cost compares favorably to existing bot mitigation — both technical and social. Cloudflare will monitor this metric closely as adoption grows.

Signing requests in practice

Cloudflare has published reference implementations on GitHub for generating Message Signatures, all standards-compliant for interoperability. For an agent built on managed Chromium, the web-bot-auth npm package works with the chrome.webRequest.onBeforeSendHeaders extension hook, which fires before HTTP data is sent and when headers are available.

chrome.webRequest.onBeforeSendHeaders.addListener(
  function (details) {
	// Signature and header assignment logic goes here
      // <CODE>
  },
  { urls: ["<all_urls>"] },
  ["blocking", "requestHeaders"] // requires "installation_mode": "force_installed"
);

The onBeforeSendHeaders hook must be implemented synchronously, so the package exposes signatureHeadersSync. Once the signature completes, both Signature and Signature-Input headers are assigned and the request continues.

const request = new URL(details.url);
const created = new Date();
const expired = new Date(created.getTime() + 300_000)

// Perform request signature
const headers = signatureHeadersSync(
  request,
  new Ed25519Signer(jwk),
  { created, expires }
);
// `headers` object now contains `Signature` and `Signature-Input` headers that can be used

The extension code, along with a debugging server at https://http-message-signatures-example.research.cloudflare.com, lets you inspect headers from the perspective of a visited website.

The demonstration stack is TypeScript end to end: the verification site runs on Cloudflare Workers, the client is a Chrome extension. To show the approach works beyond that environment, Cloudflare also implemented signature validation in Go as a plugin for the Caddy server.

A closer look at request mTLS

HTTP Message Signatures aren't the only way to cryptographically identify automated traffic. mTLS has long been used for mutual authentication between clients and servers, where both parties present TLS certificates and verify each other's private key holdings. On paper, mTLS looks attractive for bot authentication — but it has a practical problem. If a server demands a client certificate and the client doesn't have one, the user gets a hard, unskippable error with no way to signal capability beforehand.

Origins need a mechanism to tell clients, conditionally, that they accept or require mTLS authentication — without blocking ordinary traffic that lacks certificates.

A TLS flag to opt into certificate requests

Within the IETF, a proposal called req mTLS seeks to solve this using TLS flags. The approach builds on the TLS Flags draft, which lets peers exchange an array of single-bit flags instead of defining a new extension for every piece of information. The client sets the flag to indicate it can respond to a certificate request; the server, knowing the client is prepared, can safely ask for one without collateral damage to users who never set the flag.

In a Wireshark capture of the handshake, the extension appears as number 65025 (0xfe01), which sits in an unassigned block reserved for TLS Flags experimentation. The client signals support by setting the 80th bit to true within a 12-byte block, producing the value 0b0000000000000000000001. After IETF adoption, the extension number would be fixed. The server responds with a certificate request, and the normal mTLS flow proceeds.

Extension: req mTLS (len=12)
	Type: req mTLS (65025)
	Length: 12
	Data: 0b0000000000000000000001

Experimental implementation

Because mTLS itself is widely implemented in TLS libraries, the novel pieces are limited to sending and parsing TLS Flags plus handling the req mTLS flag specifically. According to the researchers, no complete public implementation of either exists yet — a gap they're hoping bot authentication work will motivate others to fill.

Using an experimental fork of Go, a client can be configured to emit the req mTLS bytes in the TLS Flags extension:

config := &tls.Config{
    	TLSFlagsSupported:  []tls.TLSFlag{0x50},
    	RootCAs:       	rootPool,
    	Certificates:  	certs,
    	NextProtos:    	[]string{"h2"},
}
trans := http.Transport{TLSClientConfig: config, ForceAttemptHTTP2: true}

Once the client sets the flag, there's nothing further to build — the standard mTLS certificate code takes over. To test an implementation, the group has published a client under cloudflareresearch/req-mtls that can be pointed at req-mtls.research.cloudflare.com.

Two paths, shared objective

HTTP Message Signatures and request mTLS both aim to give bot and agent developers a public, standardized way to prove their identity to CDNs and hosting platforms. The project is pushing both at the IETF, where TLS and OAuth Bearer tokens were also standardized through multi-stakeholder RFC processes.

The present priority, however, is HTTP Message Signatures for Bots. That effort builds on the already-adopted RFC 9421, counts several reference implementations, and operates at the HTTP layer where adoption is simpler. Request mTLS might appeal to site owners sensitive to added bandwidth, but TLS Flags has fewer implementations, hasn't yet been adopted by the IETF, and upgrading TLS stacks has historically proven harder than touching HTTP. Both schemes share the same discovery and key-management questions, which are captured in a glossary draft at the IETF.

Beyond IPs and spoofable headers

Both signature-based and mTLS-based authentication give bot owners a tamper-proof identity that doesn't depend on fluctuating IP ranges or headers like User-Agent that anyone can forge. The authentication can be consumed by a reverse proxy or by origin infrastructure directly, which lets bot operators negotiate crawling terms with content creators at whatever granularity they need — down to individual bots.

For site owners, verifiable bot identity means finer control over which automated requests to admit while keeping the public web open. Longer term, the authentication mechanisms are planned for integration into AI Audit and Bot Management products to give better visibility into agents willing to identify themselves.

Requests for verification, commercial or research implementations, or questions about use cases can be directed to the team through the verified bots program.