Why "human or bot" is the wrong question

Interaction with the online world always runs through a gateway — keyboard, screen, browser, device — and what the industry calls "human detection" is really a set of patterns humans exhibit when using those gateways. Those patterns have shifted: an executive has a browser summarize the day's news, an enthusiast scripts ticket purchases for a sale that opens at night, a visually impaired user relies on assistive accessibility features, and enterprises route employee traffic through zero trust proxies.

Site owners, meanwhile, still need to protect data, manage capacity, control distribution and stop abuse. None of those goals reduce to knowing whether the client is human: wanted bots and unwanted humans both exist. Intent and behavior are what matter — is this attack traffic, is crawler load proportional to the traffic it returns, do I expect this user from this country, are my ads being gamed? Detecting automation remains important, but systems should be built for a future in which the bot/human split is no longer the key data point.

The word "bots" actually covers two separate problems. The first is whether operators should admit known crawlers that send no traffic back; bot authentication with HTTP message signatures lets such crawlers identify themselves without being impersonated. The second is the rise of clients that do not share the behaviors web browsers historically exhibited — a distinction that matters for mechanisms such as private rate limiting.

How the browser-mediated Web held together

Users do not talk to the thousands of servers they touch each day directly; they go through browsers, or user agents, which act on their behalf so that sites never get access to an entire computer or phone. Sites in turn care how browsers behave: content must render correctly, purchases and passwordless sign-in must complete, microphones must work, ads must be seen.

That tension between publisher control and user agency is old, and browser vendors and the standards ecosystem have weighed it deliberately. Extensions can block ads, though browsers have progressively restricted what such extensions may do. Accessibility standards such as WCAG legitimized non-pixel uses of content, backed in many jurisdictions by regulation. Whether one likes each tradeoff, they come as a package: participation on the Web means accepting it.

AI is now shifting that balance. An assistant that summarizes or aggregates is not a new idea, but AI makes the capability available to everyone. The friction lies in how these clients operate. A human assistant might print an article or take a screenshot without the publisher knowing, but still renders the site in a standard browser first. AI agents skip that step and quietly fetch raw data without rendering the page. Because their traffic overlaps with pre-existing browser traffic, publishers cannot tell whether a fetch serves one private report — possibly distorted, possibly unattributed — or feeds a model trained for a million users, undermining the predictable, monetizable traffic that funds the site.

The client-server model and its blind spot

The Web's core deployment pattern is simple: a client requests a resource and a server responds.

BLOG-3114 Image 1

Figure 1: Client-Server model. A client sends a request which the server responds to.

Scaling works on both sides. A site adds servers or puts a cache in front of static traffic; request volume grows when one client asks for more or when clients multiply.

BLOG-3114 Image 2

Figure 2: Multiple clients send multiple requests to different servers, with one fronted by a CDN.

That simplicity helped the Web succeed: many kinds of clients can exist and the network evolves without servers knowing what software sits at the other end.

BLOG-3114 Image 3

Figure 3: Two different client contexts that send requests to servers. Each server only sees a request, but not the end-user behind it.

Openness breeds uncertainty. A server sees a valid request but generally cannot know what happens after the response leaves — whether it is rendered for a person at a keyboard, mouse and screen, or consumed by a program that archives, indexes and feeds it into a larger system.

What bot management actually does

The model works well until a server must decide which requests it can afford to serve, trust or prioritize. Capacity is one driver: provisioned for 100 requests per second but receiving 200, running one CPU against demand for two, or facing prohibitive cost for serving 200 all force drops or rate limits. Random dropping is unfair and can hit wanted clients, but without other signals there is no alternative.

Capacity is only part of it. Servers also separate attacks from ordinary traffic, manage benign load, prevent data extraction, limit ad fraud, block fake account creation and stop automated actions taken on a user's behalf. Web clients are unauthenticated by default yet expose many partial signals, so access logic is built on what arrives. One IP making ten times the requests of others might be blocked; an IP inferred to be a VPN exit proxying multiple users might receive a coefficient — at ten requests per second per client, a hundred before drops.

Bot management exists to give the server more information about the client, and that information is inherently imprecise because the client is outside the server's control. The same data also forms fingerprint vectors usable for other ends, such as personalized advertising, turning a mitigation vector into a tracking vector.

At a high level the server sees three classes of signal:

  1. Passive client signals: required to make a request at all — the IP address, and usually a TLS session.
  2. Active client signals: volunteered by the client, often invisibly, such as a User-Agent header or authentication credentials.
  3. Server signals: what the server itself observes, such as the geography of the edge handling the request or local receipt time.

Against volumetric abuse, what matters to the origin is the client's capability and intent to make repeated requests. An ad-funded site needs confidence that ads reach an end user; brand protection may require specific rendering capabilities such as a PDF reader, SVG renderer or virtual keyboard; a request arriving through an intercepting proxy may need to be shown to originate from an end client. Costs scale with traffic, and traffic that generates no value — monetary or otherwise — gives the server no reason to bear those costs.

Operators respond differently. Large crawlers and platforms often identify themselves because predictable access is worth being attributable, and it can even help. Others evade identification because they expect blocking, seek anonymity or act on behalf of end users. The result is an unstable equilibrium resting on partial signals — and the reason the human-versus-bot frame misleads: what origins care about is whether a client behaves in ways the site can support.

The rate-limit trilemma

BLOG-3114 Image 4

Figure 4: Rate limit trilemma. Decentralized, anonymous, accountable — pick two

Access governance faces a fundamental tension: decentralized, anonymous and accountable — pick two.

  1. Decentralized + anonymous leaves no accountability. A blocked client spawns a new account with no reputational cost, so origins must invest more in managing resources. This is the Web's default.
  2. Decentralized + accountable means everyone knows who you are. It suits some use cases but has clear drawbacks — OAuth-style "Log in with" mechanisms require registration and reveal activity to a third party.
  3. Anonymous + accountable implies governance, rules and enforcement. No widely deployed system delivers both properties for the same actor. The nearest precedent is the Web PKI, where CA policy and Certificate Transparency hold servers accountable, and failures of that governance have consequences. Nothing equivalent exists on the client side today.

Current tools draw on the first space while reaching for the second: TLS fingerprints, IP addresses, robots.txt. They aim at accountability but hold only as long as the derived fingerprints stay stable.

Self-identifying infrastructure versus everyone else

For a site operator, the useful split isn't bots versus humans. It's traffic that wants to be recognized and traffic that needs to stay anonymous — and the origin has to balance its own need to understand requests against clients' need to keep their privacy.

Operators who can afford attribution

BLOG-3114 Image 5

Figure 5: A crawler makes multiple request to a server

Search crawlers, cloud platforms and enterprise infrastructure generate high request volumes from known sources and generally have low privacy expectations. Knowing where a request came from lets an origin correct course when a provider sends too many requests or fetches pages it shouldn't. Self-identification is one of the principles for responsible AI bots, and it underpins how Cloudflare runs its URL scanner for Radar and exposes its crawling capabilities.

Identity works for this traffic because reliable access is worth being attributable. With Web Bot Auth built on HTTP Message Signatures, operators can cryptographically sign their requests; OpenAI, Google, Cloudflare and AWS already sign requests originating from their platforms. Origins can then verify that a request really came from the platform's infrastructure instead of trusting IP ranges or User-Agent strings.

Everyone else: many sources, few requests each

BLOG-3114 Image 6

Figure 6: Three distinct browsers make a request to a server. One is operated by a human, one by an on-device assistant, and one is proxied through a corporate proxy.

Other traffic arrives from many sources making relatively few requests each: people browsing, researchers taking measurements, scrapers behind residential proxies, and increasingly AI assistants acting for humans. Here the bot/human distinction stops mattering. An AI assistant booking concert tickets is indistinguishable in kind from the human who would have done it manually — both are distributed, both need anonymity, and in both cases the origin wants low friction for those using the service as intended rather than abusing it.

Identity could in principle carry the load, provided it meant a unique, verifiable set of attributes tied to a specific client and proven through a login, an email address or a hardware key. But that implies presenting that identity to every website, which undermines privacy. The better target is proving behavior without proving identity.

Privacy Pass and its limits

Clients reaching websites through Cloudflare have been able to send a privacy token since 2019, when Cloudflare added support for Privacy Pass. Standardized in RFC 9576 and RFC 9578, it lets a client carry an issuer-backed proof of an earlier check — solving a challenge, say — without that result becoming a stable identifier. Its tokens are unlinkable to any prior visit, request or session. Rather than harvesting passive signals as fingerprinting does, a server asks the client for an active, privacy-preserving one. Privacy Pass has scaled to billions of tokens per day across Cloudflare's infrastructure, mainly for privacy relay services.

BLOG-3114 Image 7

Figure 7: Privacy Pass Redemption and Issuance Protocol Interaction from Section 3.1 of RFC 9576

The RFC defines four roles. An issuer trusts one or more attesters to run checks before issuing credentials. The client holds the credentials and chooses when to present them, within scope. The origin decides which issuers it trusts and what a presentation means. Abuse and policy questions remain; what changes is that client and server get a privacy-preserving way to handle them.

The design has three concrete bounds:

  • No dynamic rate limits. If a client is issued 100 tokens and starts consuming too many resources after a session or two, the remaining tokens can't be invalidated.
  • New issuers struggle to emerge. Unlinkability leaves no feedback channel through which an origin can signal the quality of a token's issuer.
  • Presentations are 1:1 with tokens. One token buys one unlinkable presentation. What's wanted instead is a client that contacts an issuer once and later makes multiple presentations scoped to a given origin — user agents holding vouched credentials and deriving proofs from them rather than repeatedly acquiring single-use tokens.

Toward an open private rate limiting ecosystem, Cloudflare is helping develop new Privacy Pass primitives: Anonymous Rate-Limit Credentials (ARC) and Anonymous Credit Tokens (ACT). ACT lets a client prove something like "I have a good history with this service" without revealing "I am this user," preserving unlinkability between presentations at the protocol level. Even in the joint issuer-origin model of RFC 9576 Section 4.3, issuance and presentation aren't directly linkable by design — though correlation through IP addresses, cookies, account state or timing is not eliminated. ACT implements these properties with the standardized VOPRF and BlindRSA primitives inside a reverse flow framework.

An open issuer ecosystem means more than letting anyone mint credentials. Origins must be able to choose which issuers to trust, user agents need a consistent way to present what is being requested, issuers need a path to reputation, and relying parties need a way to drop low-quality issuers. No single gatekeeper should control participation. That in turn requires a protocol and client API spanning browsers and other user agents: simple to deploy, clear to users, and narrow enough that browsers can cap abusive proof requests instead of merely surfacing them.

What happens without a fix

Site owners are already responding to disruption from new kinds of clients, driven partly by large-scale scraping and model training and partly by agents behaving in ways sites did not anticipate. The demand for technical means to block AI crawlers follows from that. As the line between bots and humans blurs, today's measures weaken on their own.

Sites would then pivot to requiring an account for any content, or tying access to a stable identifier — the end of ad-supported, login-free articles and of "three free articles a month." Some content businesses may leave the Web entirely, selling data and services directly to AI vendors or operating inside large platforms' walled gardens.

Not every site would make those choices; content goes online for many reasons, not all commercial. But if enough sites do, they reset what counts as normal on the Web, and for the worse. The open Web lets many different clients gather information from many different sources rather than a handful of players, and a diversity of sources matters in its own right. If access is largely mediated by a few companies, too much power sits in too few hands — and the result is more friction for anonymous clients and a more brittle Internet with fewer routes for publishers to reach users.

Anonymous proofs can be repurposed

The same infrastructure that lets a user prove a property can be turned into infrastructure that requires one. Anonymous credentials exist to attest to a single thing about the holder — "I solved a challenge," "I have not exceeded a rate limit" — but any system able to prove one attribute can be extended to prove others, and that is where the risk sits.

Today a Privacy Pass token may signal that someone solved a CAPTCHA. Tomorrow the same machinery could demand unrelated attributes: tokens issued only to devices carrying device attestation would cut off older hardware and its users, and requiring something like an "has an Apple or Google account" attribute would exclude anyone on a non-mainstream platform. Once verification of anonymous proofs is in place, the set of things proven can keep growing, so the important constraint is that this does not become a gate on Internet access.

The alternative is fingerprints

Gates are already here — platforms demanding identity, sites blocking traffic from shared proxies — so the real question is not whether they appear but whether the user keeps control of their privacy. Bot management needs shared signals either way. What anonymous proofs replace is worse: without them, every gate falls back to fingerprints — retry from a specific browser, link your account, don't use a VPN. For some people those aren't even available, including those who have no idea their connections are proxied.

Privacy-preserving credentials neither remove trust nor replace policy; they make demands explicit rather than pervasive. A proof is visible, so users can see what is being asked, and clients such as browsers and AI assistants can help enforce consent.

Guardrail for new methods

One test should decide each proposed method: can anyone, anywhere, build their own device, run their own browser, use any operating system, and still reach the Web? If device attestation from particular manufacturers becomes the only viable signal and that property fails, the method should be dropped.

Holding that line means an open issuer ecosystem in which no single gatekeeper decides who participates. Within the rate limit trilemma, decentralization is mandatory for the open Web, even though how to build it isn't fully known yet — only that it must be fostered.

A balance worth keeping

The Web has stayed roughly in balance so far, partly by accident and partly by necessity: it worked for most users and publishers because it remained open enough for a wide range of clients to reach a wide range of resources. That balance is now at risk.

Privacy-preserving primitives for the Web are an attempt at a different outcome — private, open and accountable at once. Success isn't guaranteed, but it beats waiting. The work happens in the open at the IETF and the W3C, and those existing venues, where the Web of today was shaped, remain the best places to design the Web of tomorrow. The Internet is for the end user, and the end user belongs at the center of it.