Crawler Hints Hits General Availability with IndexNow Support
Cloudflare has announced that Crawler Hints, its service for reducing wasted web crawler traffic, is now generally available to all customers at no cost. The service has also added support for IndexNow, a protocol developed by Microsoft and Yandex that lets websites notify search engines about content changes.
Crawler Hints addresses a significant inefficiency in how search engines discover updated content. Search engine bots must continuously revisit websites to detect changes, but without a central system for tracking content updates, they rely on estimates of how often a site changes. Cloudflare's analysis indicates that roughly 53% of crawler traffic is wasted on revisits to pages that have not changed since the last crawl.
Because Cloudflare proxies a substantial portion of web traffic, it can observe when content on an opted-in site actually changes. Crawler Hints uses this visibility to push signals to search engines about when a site should be recrawled, replacing the need for frequent speculative bot visits.
What IndexNow Adds
IndexNow is an open protocol created by Microsoft and Yandex that provides a standardized way for websites to ping search engines when content is added, updated, or deleted. With Crawler Hints now supporting IndexNow, Cloudflare handles these notifications automatically for customers who enable the feature. There is no additional configuration required beyond turning on Crawler Hints in the dashboard.
For search engine operators, the benefit is a near real-time, pushed feed of content-change events from opted-in Cloudflare sites. This allows crawlers to prioritize fresh content and reduce unnecessary requests, which improves both result quality and the energy efficiency of bot infrastructure.
Industry Support
Cloudflare says it has been testing IndexNow integration with Microsoft Bing and Yandex for months with early success. The company also notes that it is open to working with other organizations that rely on crawling to power their services.
"The cache data from CDNs is a really valuable signal for content freshness. Cloudflare, as one of the top CDNs, is key in the adoption of IndexNow to become an industry-wide standard with a large portion of the internet actually using it. Cloudflare has built a really easy 1-click button for their users to start using it right away. Cloudflare's mission of helping build a better Internet resonates well with why I started IndexNow i.e. to build a more efficient and effective Search."- Fabrice Canel, Principal Program Manager
"Yandex is excited to join IndexNow as part of our long-term focus on sustainability. We have been working with the Cloudflare team in early testing to incorporate their caching signals in our crawling mechanism via the IndexNow API. The results are great so far."- Maxim Zagrebin, Head of Yandex Search
"DuckDuckGo is supportive of anything that makes search more environmentally friendly and better for end users without harming privacy. We're looking forward to working with Cloudflare on this proposal."- Gabriel Weinberg, CEO and Founder
Benefits for Site Owners
For Cloudflare customers, enabling Crawler Hints means search engines are more likely to index the freshest version of their content, which can influence search rankings. The service also reduces the volume of crawler traffic hitting origin servers, lowering resource consumption and freeing bandwidth for human visitors.
End users benefit indirectly: bot-fed experiences such as search engines and pricing tools will deliver more current results, since the underlying crawled data is refreshed at the moment content changes rather than on an arbitrary schedule.
Enabling Crawler Hints
Crawler Hints is free for all Cloudflare customers. To enable it:
- Sign in to your Cloudflare account.
- In the dashboard, navigate to the Cache tab.
- Click on the Configuration section.
- Locate the Crawler Hints sign-up card and enable it.
Once enabled, Cloudflare begins sending hints to participating search engines about when specific parts of the website should be crawled. Cloudflare describes the rollout as an ongoing effort and expects to add more partners over time as the industry moves toward more efficient crawling practices.
