Discovery is the missing piece for signed bot traffic

As bots and agents begin cryptographically signing their requests, website operators face a new problem: how do they learn the public keys needed to verify those signatures? Finding key material for well-known fetchers and crawlers is feasible, but what about the next thousand, or the next million? And once found, how does an operator confirm the key actually belongs to the party it claims to represent? That problem is discovery.

Cloudflare shares this problem with Amazon Bedrock AgentCore, whose AgentCore Browser is a cloud-based browser runtime for AI agents interacting with websites at scale. The AgentCore team wants each of its customers to sign their own requests, so CDN operators see signatures from individual agents rather than from AgentCore as a monolith. That requires a way to ingest and register the public keys of AgentCore's customers at scale.

The proposal is a registry of bots and agents, described in the draft specification — a list of URLs at which agent keys can be retrieved, authored and imported as easily as an IP address list. The goal is an open ecosystem of curators that website operators can choose to trust.

The verification gap

Web Bot Auth, introduced in May, lets bot developers cryptographically sign requests from their infrastructure. Multiple implementations now exist — from Vercel and Shopify to Visa, and it has been discussed in IETF mailing lists. But cryptographic keys, like IP addresses, are a pseudonymous form of identity. For a website operator without the scale of a large CDN, discovering the public keys of known crawlers remains difficult.

The initial protocol proposal suggested one approach: bot operators provide a Signature-Agent HTTP header referencing an endpoint that hosts their keys. As with IP addresses, the default is to allow all traffic, but when a particular operator misbehaves you can take action — increase their rate limit, contact them, or otherwise restrict them. An example can be seen in Shopify's online store:

Signature-Agent: "https://shopify.com"

A list of key endpoints

The ecosystem for IP lists (like avestel-bots-ip-lists) and robots.txt files (like ai-robots-txt) works because canonical lists are publicly available and easily imported into nginx, haproxy, or a Cloudflare account. The registry format replicates that model for cryptographic keys — a plain list of URLs pointing at Signature-Agent keys.

User-agent: MyBadBot
Disallow: /

The registry itself is just that list of endpoints:

# AI Crawler
https://chatgpt.com/.well-known/http-message-signatures-directory
https://autorag.ai.cloudflare.com/.well-known/http-message-signatures-directory
 
# Test signature agent card
https://http-message-signatures-example.research.cloudflare.com/.well-known/http-message-signatures-directory

A registry may contain all known signature agents, a curated list for academic researchers, one for search agents, and so on. Anyone can host one on any public file system — a GitHub repository, Cloudflare R2, or even an email attachment. Cloudflare intends to run one of the first instances so others can contribute to it or reference it in their own setups.

Attaching context to a key

A signature proves the request came from a particular key, but not much else. Verified bot operators, for example, need a contact method in case their requests suddenly change format and break something upstream. Origins may also want to know the operator's name, a logo, or the expected crawl rate.

To support that, the specification includes a signature-agent card format that extends the JWKS directory (RFC 7517) with additional metadata. The card holds all the relevant information about an agent, much like a contact card. Fields may evolve — for instance, introducing jwks-uri or making the logo more descriptive.

{
  "client_name": "Example Bot",
  "client_uri": "https://example.com/bot/about.html",
  "logo_uri": "https://example.com/",
  "contacts": ["mailto:[email protected]"],
  "expected-user-agent": "Mozilla/5.0 ExampleBot",
  "rfc9309-product-token": "ExampleBot",
  "rfc9309-compliance": ["User-Agent", "Allow", "Disallow", "Content-Usage"],
  "trigger": "fetcher",
  "purpose": "tdm",
  "targeted-content": "Cat pictures",
  "rate-control": "429",
  "rate-expectation": "avg=10rps;max=100rps",
  "known-urls": ["/", "/robots.txt", "*.png"],
  "keys": [{
    "kty": "OKP",
    "crv": "Ed25519",
    "kid": "NFcWBst6DXG-N35nHdzMrioWntdzNZghQSkjHNMMSjw",
    "x": "JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs",
    "use": "sig",
    "nbf": 1712793600,
    "exp": 1715385600
  }]
}

Who runs a registry

Amazon Bedrock AgentCore has adopted Web Bot Auth for its AgentCore Browser service. During the public preview, AgentCore Browser uses a single service signing key, but plans to transition to customer-specific keys as the protocol matures. That would let Cloudflare and other origin protections see signatures originating from individual AgentCore customers rather than a single aggregate key.

Cloudflare already provides a registry of bots and agents it trusts via Radar, using the registry format. A Go demo for Caddy server that imports keys from multiple registries is available in a pull request on cloudflare/web-bot-auth:

:8080 {
    route {
        # httpsig middleware is used here
        httpsig {
            registry "http://localhost:8787/test-registry.txt"
            # You can specify multiple registries. All tags will be checked independantly
            registry "http://example.test/another-registry.txt"
        }

        # Responds if signature is valid
        handle {
            respond "Signature verification succeeded!" 200
        }
    }
}

Operating and curating a registry with the Signature Agent Card format offers several advantages:

  1. Monitor incoming Signature-Agents. You can collect signature-agent cards from agents reaching your domain.
  2. Import and categorize the lists yourself. A general registry may come from a monitoring step, but narrower categories make a registry more useful.
  3. Establish direct relationships with agents. Cloudflare does this for its bot registry, or you could run a public GitHub repo where operators file issues.
  4. Learn from your users. If you offer a security service, letting customers pick which registries and signature-agents to allow gives you insight into what they value.

Toward a curated trust model

As cryptographic authentication for bots grows, so does the need for discovery. The proposed lightweight format attaches metadata to Signature-Agent and organizes agents into registries. The HTTP Message Signature directory format is being expanded with self-certified metadata, and registries provide the curation layer.

The likely trajectory: clients and origins select which signature-agent registries they trust, use a common format to move configuration between CDN providers, and rely on third-party registries for curation. The demo is available on GitHub for those who want to experiment with importing keys from multiple registries.