A Push to Keep Search Fresh
When Cloudflare first floated Crawler Hints in 2021, the pitch was simple: give search engines better intel on when web content actually changes. Instead of crawling on fixed schedules that often miss real-world updates, the idea was to let the people who see traffic firsthand — Cloudflare, sitting in front of millions of sites — tell indexers what’s stale. For site owners, that means better SEO and less bot traffic hammering their origins. For search engines, it means fewer wasted crawls. For the planet, it means a bit less energy burned on redundant requests.
To make that work, Cloudflare taps into signals it already sees at the edge. The system watches for cache misses on opted-in sites: when a user requests content that isn’t in the local data center cache, that’s a reasonable hint that the origin has something new or changed. A cache miss on its own isn’t a perfect freshness signal, but it was the right starting point. The key was getting the plumbing in place — pushing miss data from Cloudflare’s network into a pipeline that could scale to the company’s traffic volume: over 28 million HTTP requests per second on average, peaking above 35 million.
Unlike most Cloudflare products that sample traffic, Crawler Hints can’t afford to drop a single cache miss. If a page is newly available, that miss needs to be counted. So the decision was made to forward 100% of cache misses for opted-in zones into the processing pipeline. Request data gets marshalled into Cap'n Proto format and pushed to Kafka, carrying the URL of each fresh resource along with metadata for analytics and future signal refinement.
Deduplicating at Scale with Redis
The next problem is one of aggregation. Kafka buffers the incoming misses, but what arrives is raw and noisy. A consumer service — the “ingestor” — pulls from Kafka, validates and sanitizes the data, and moves it toward the next stage. The ingestor runs as part of a Kafka consumer group, so Cloudflare can scale the number of consumers up as throughput climbs.
But the real engineering challenge is deduplication. The system wants to group fresh URLs into batches — say, 10,000 URLs every two minutes — without repeating the same resource twice in a single batch. An in-memory map would be the obvious shortcut, but it has a fatal flaw: if the service crashes, the entire batch state is lost. Run multiple instances and they’d each have their own partial view, defeating the whole purpose of deduplication.
Cloudflare’s answer is Redis, used not as a cache in the conventional sense but as a distributed rolling buffer. Two Redis clusters stand by, with one designated active for the current generation. As validated records stream in, they’re written as keys into the active cluster — keys are unique, so insertion itself is the deduplication. Periodically, a downstream service flips the active role from cluster A to cluster B, then flushes A by reading out its records in appropriately sized batches.
Don’t Drop the Batch on the Floor
Clean, batch-read data is promising — but there’s a durability gap. If the dispatcher interval runs faster than the partner API can ingest, or if that API goes down entirely, those fresh-URL records can’t just vanish. A crash between Redis reads and a successful HTTP call upstream shouldn’t mean lost signals.
The solution is to reintroduce Kafka on the outbound side. Batches read from Redis feed into a dedicated Kafka topic, and a “dispatcher service” consumes it as part of a consumer group — same scaling pattern as the ingestor. The dispatcher takes each batch and fires it at the search partners.
That partnership model is largely defined by IndexNow, an open protocol launched alongside Crawler Hints. IndexNow provides a standard API spec for publishing URLs that need re-indexing, which in turn makes it clean for Cloudflare to abstract the communication layer across partners. Both Bing and Yandex signed on as early adopters — a push-based model replacing an inefficient pull-based one.
For future extensibility, Cloudflare defined a “Hinter” interface that each partner implements. Cloudflare handles errors with a custom type that records which indexer failed, allowing targeted retries and metrics instead of a blanket resend to every partner.
Operating Crawler Hints at Scale
Because Crawler Hints is a new type of service, predicting customer interest and system load in advance is difficult. Engineering for scale meant building safeguards to prevent the system from being overwhelmed, while ensuring it could handle unexpected growth. Three layers of protection were built in: customer opt-in, comprehensive monitoring and alerting, and self-healing infrastructure.
Customer Opt-in
Cloudflare treats any change that could affect customer traffic with caution. Since Crawler Hints changes how websites appear to crawlers, which can influence SEO and bandwidth, we made the service opt-in. This approach also lets us observe real-world capacity and identify bottlenecks early. Monitoring relies on Prometheus, Grafana, and Kibana.
Monitoring and Alerting
Even with self-healing systems, failures happen when least expected. Automated health tracking and alerts are critical. The Grafana dashboards show metrics like customer enablement rates and hint dispatch speed, along with Kafka cluster throughput by partition. Other tracked metrics include:
- Kafka lag per partition
- Malformed messages from Kafka
- URLs processed per run
- Response codes from each index partner
- Partner API response times
- Redis cluster memory usage and command rates
- CPU, memory, and pod availability versus limits
Alerts notify the on-call engineer when thresholds are breached. A Redis cluster nearing 80% capacity would trigger a page, since a human may need to decide whether the cause is a misconfiguration or simply that the product grew in popularity.
Self-Healing Infrastructure
The goal is to minimize disturbances to on-call engineers while avoiding over-provisioning, which is expensive and wastes capacity other Cloudflare services might need. Running on Kubernetes makes Horizontal Pod Autoscaling available: pods spawn automatically when memory usage approaches 80%, up to a configurable limit.
The message bus provides control over work volume, since it is pull-based. However, because near-real-time operation matters, monitoring topic lag is essential. If lag grows too large, adding partitions or consumers may be necessary.
Network failures are unavoidable, so all HTTP calls include retry policies. A 500 response from a partner API is retried up to five times with exponential backoff before being reported as a failure.
Launch Metrics
Between the October 18, 2021 launch and December 15, 2021, Crawler Hints processed over 25 billion crawl signals for more than 81,000 opted-in customers, sustaining roughly 18,000 requests per second.
Roadmap
Future work includes refining the signaling standard with partners and improving how the most valuable information is pushed to them in a timely manner.
