Search Indexing Still Works Like It's 1999
Search engines have indexed the web the same way for decades: they dispatch crawler bots on a scheduled "pull" model, revisiting sites at set intervals to check for changes. That constant polling adds up — bot traffic accounts for roughly 30% of all internet traffic, and Cloudflare estimates that over half of crawler activity is wasted effort. Every unnecessary crawl costs website owners in bandwidth, CPU, and infrastructure capacity, and at internet scale that adds a significant environmental toll.
The inefficiency is baked into the architecture. Sites must over-provision to absorb bot floods, and content can sit unpublished in search results until the next scheduled crawl arrives. Meanwhile, search engines burn resources revisiting pages that haven't changed. A push model — where sites alert engines the moment content updates — would solve much of this, but it requires coordination on both ends: sites need a signaling mechanism, and engines need a way to receive those signals and adjust their crawl schedules accordingly.
Crawler Hints Fills the Signaling Role
Cloudflare's Crawler Hints, launched in October 2021, tackles the site side of the problem. Because Cloudflare terminates traffic for a large share of the web — currently averaging 36 million HTTP requests per second — it can observe origin status codes and cache behavior across millions of sites. Crawler Hints uses that data to notify indexers when content has likely changed or been added, giving crawlers a reason to revisit a page instead of polling it blindly.
The service aims to boost the ratio of productive crawls — those that actually find fresh content — while cutting the wasted requests that return nothing new. For site operators, that means less load from bots; for indexers, lower crawl costs; and for users, fresher search results. The feature is free for all Cloudflare customers and can be enabled in the dashboard under the Cache tab, in the Configuration section. Once switched on, Cloudflare handles the signaling automatically via the IndexNow API — no additional setup required.
IndexNow Gives Engines a Push-Based Ear
Signals are only useful if search engines are listening. The IndexNow protocol, developed by Microsoft, Seznam.cz, and Yandex, provides that receiving end. It gives search engines a standard interface to accept change notifications and shift their crawlers from a pull model to a push model. When a site pings IndexNow, participating engines can schedule a targeted recrawl rather than waiting for their next general sweep.
The protocol has gained real traction. Microsoft reports that millions of sites now use IndexNow, and URLs submitted through it account for about 7% of all new URLs clicked from web search results. On Cloudflare's side, Crawler Hints has processed roughly 600 billion signals to IndexNow since launch — a substantial volume of avoided crawls.
Planned Expansion
Cloudflare intends to enable Crawler Hints by default for all customers, given the benefits to site owners, search operators, and the environment. The company is also advocating for broader industry adoption, hoping Google — the largest driver of crawl traffic — will embrace IndexNow or a comparable standard.
Future iterations of Crawler Hints will incorporate additional change-detection signals, including etags, last-modified headers, and content hash differences. These would give crawlers a more precise picture of when a site actually needs re-indexing and how frequently they should return. The work is ongoing, but the core shift is already in place: making crawl signals explicit instead of leaving search engines to guess.
