Project Galileo Adds AI Crawler Controls for Newsrooms
Cloudflare is extending its Project Galileo program with free access to Bot Management and AI Crawl Control for participating news organizations. The program, launched in 2014 to protect civic voices online, currently serves roughly 750 journalists, independent news organizations, and non-profits supporting news-gathering worldwide. It now includes more than 3,000 organizations across 125 countries.
The additional tools are aimed at addressing a growing problem for publishers: the shift in web traffic from search engines to AI models. As users increasingly get answers directly from AI systems, fewer of them click through to the original source. That matters for newsrooms that depend on reader traffic for revenue from subscriptions, advertising, e-commerce, and licensing. Cloudflare’s stated goal is to give these organizations the means to monitor AI crawlers, control access to their content, and negotiate fair compensation for their work.
Why News Organizations Are Under Pressure
Cloudflare's own data highlights the scale of the challenge. The company reports that with OpenAI, it is 750 times harder for website owners to get the same volume of traffic as they did from Google search previously. With Anthropic, that figure rises to 30,000 times more difficult. The Financial Times' CEO has cited a “pretty sudden and sustained” decline of 25% to 30% in traffic to its articles arriving via search engines.
This comes as the broader news industry is already fragile. According to recent US Congressional testimony, more than two newspapers closed per week in 2024, with one third of the country's newspapers projected to close before the beginning of 2025. A 2024 report from the Northwestern Local News Initiative found more than 206 US counties had no local news source at all, and 1,561 had only one. Funding cuts to the Corporation for Public Broadcasting and National Public Radio have added further strain, with more closures expected.
Monitoring and Controlling AI Access
Bot Management uses machine learning to analyze web traffic and distinguish between helpful bots—such as search engine crawlers—and harmful ones that attack sites or steal credentials. Site owners can block the latter while allowing useful bots to continue functioning.
AI Crawl Control applies similar techniques specifically to AI crawlers. Cloudflare identifies and categorizes these crawlers using HTTP headers, heuristics, and behavioral analysis. That allows website owners to examine activity by type (AI search, AI scraper), by origin (Google, OpenAI, Anthropic, etc.), and by the content being accessed.
“In an era defined by AI and digital disruption, providing robust tools to independent media isn’t just support - it’s a lifeline” - Meera, CEO Internews Europe
The controls are granular. Site owners can allow or block access crawler-by-crawler, meaning a local newspaper could, for example, block all AI crawlers to maintain a direct connection with readers, block only AI scrapers while permitting AI search crawlers that refer traffic, or negotiate exclusive access for a single AI company.
Cloudflare positions these capabilities as both an immediate safeguard and a longer-term strategic asset. Monitoring crawler activity provides insight into how AI models are using a site's data, which can inform negotiations with AI companies over time.
"Independent media's ability to fulfill its democratic function by gathering news and distributing trusted information depends on generating revenues free from political or business influence. By monitoring and monetizing the crawling of publisher's sites, media can protect their intellectual property while developing new revenue streams to support their quality journalism." - Ryan Powell, Head of Innovation and Media Business at International Press Institute
"Independent publishers need tools that are easy to use and affordable, so they can focus on growing their business. LION appreciates the security and protection Cloudflare has provided our members through Project Galileo for years, and we're excited to see more resources now available to help members manage the rapidly evolving landscape of digital security." - Sarah Gustavus Lim, LION Membership Director
The Stakes for Local Journalism
Journalism underpins democratic governance by holding power accountable and giving a voice to the marginalized. Local news, in particular, fosters shared community identity through coverage of events, schools, and local officials, while providing oversight over those in power. Cloudflare views support for journalists as central to its mission and to the future of the Internet—Project Galileo has also served as the template for other free programs protecting democratic elections, public schools, public health clinics, and critical infrastructure.
