Cloudflare and OpenAI began a research pilot in July 2026 to test whether signals from Cloudflare's network could help ChatGPT users find content more efficiently. The two companies have now published early results: the pilot has brought previously unknown pages into OpenAI's search index, expanding the pool of sources ChatGPT can draw on. OpenAI independently evaluates and ranks those pages by relevance and helpfulness rather than accepting Cloudflare's signals as a ranking input.
What the pilot covered
The test ran on a participating cohort of roughly 1% of Cloudflare's total network. Over that period, OpenAI discovered and indexed hundreds of millions of new web pages. 
The pages at stake are often unglamorous but high-value: the manual for a specific washing machine model, current passport requirements and fees, a restaurant's latest menu and opening hours. Such pages can matter even when few other sites link to them. ChatGPT Search links back to relevant sources, giving users a path to the original content and the publishers behind it. OpenAI's GPT-6 and Intelligent UI work, which makes ChatGPT more visual and interactive for shopping, learning and answers, complements these discovery improvements by widening the information available to ChatGPT.
Judging a URL before fetching it
Finding a URL is only the first step. Two questions follow: can someone actually reach the page, and does it offer anything distinct?
Automated traffic can make a nonexistent page look interesting. Bots repeatedly request files that aren't there; some sites redirect nearly any address to their homepage; other addresses resolve to duplicate content. Detecting these patterns lets the crawler skip wasted work before it fetches a page, which reduces load for site owners.
Cloudflare and OpenAI are also testing signals that flag potentially harmful pages. A single compromised page on an otherwise legitimate site — a fake login form, for example — can be flagged on its own instead of penalizing the whole domain. Early results suggest these signals can help keep such pages out of an index.
Crawl efficiency and the new discoverability
Under the older web model, discoverability meant ranking high in search results, and the human who landed on a page decided what it was worth. As people delegate more finding and task completion to AI, discoverability takes new forms: systems must locate the right sources, assess their usefulness, and connect people to them through an answer, a citation, or a visit.
Fetching every page more frequently would be expensive on all sides — for a site owner, a crawler repeatedly downloading an unchanged page adds load with no benefit. The filtering signals keep crawlers away from nonexistent or redirecting addresses, while change signals indicate when a page is worth revisiting and when it isn't. There is a tension to manage: discovering new pages can initially increase crawling as those pages are evaluated and indexed. The aim is to keep finding and refreshing content while reducing the work required.
Boundaries and next steps
A signal about a page does not grant permission to access it, and domain owners' choices remain in force throughout. The research uses metadata about the web for search and discovery; it does not involve sharing page content for model training. Site owners who want to manage how AI systems reach their sites can use Cloudflare's AI Crawl Control.
The next phase will refine page selection, test faster and more reliable change signals, and measure what happens after content is discovered. Cloudflare and OpenAI say they will continue studying how improved discovery translates into business outcomes for publishers and site owners, and will share findings as the work continues. 
