Cloudflare is pushing regulators to look more closely at how Google gathers web content for search and AI. The company’s chief executive, Matthew Prince, says the core issue is not simply whether AI companies should pay for content, but whether Google’s position in search gives it a different kind of access from the rest of the market.
Prince is in London to speak with the U.K.’s Competition and Markets Authority, where he is proposing stricter rules for Google as AI products become more closely tied to web discovery. His case centers on Google’s crawler, Googlebot, and whether site owners should have a clean way to separate ordinary search indexing from AI use.
The regulatory opening
The U.K.’s Competition and Markets Authority earlier this month designated Google with a special status in the search and advertising markets because of its “substantial and entrenched” position. That decision gives the regulator room to consider rules that go beyond classic search and ads.
The areas named in the source include Google’s AI Overviews and AI Mode, along with the Discover feed, Top Stories, and News tab. For Cloudflare, that matters because the boundary between search distribution and AI products is becoming harder for publishers and businesses to manage.
Cloudflare’s position is also shaped by its own role on the web. Earlier this year, the company launched a marketplace that lets websites charge AI bots for scraping their content. That move placed Cloudflare in the middle of a growing conflict between websites that produce content and AI companies that want to use it.
Prince argues that Cloudflare can offer useful recommendations because it is not itself trying to build a competing AI system and is not a media publisher. At the Bloomberg Tech conference in London this week, he said Cloudflare sits between those groups and added that 80% of the AI companies are Cloudflare customers.
Why Googlebot is the focus
The dispute is about bundling. According to Prince, Google uses its existing web crawler for its search engine and also for AI products and services. He says that creates a practical advantage that other AI companies do not have.
Other companies, including Anthropic, OpenAI, and Perplexity, may have to pay for content that Google can reach through the same crawler already tied to search. Prince’s concern is that this structure makes Google’s access to web content look less like ordinary competition and more like a benefit created by its search dominance.
A Google spokesperson, Ned Adriance, said site owners can use Google Extended to opt out of having content used for training AI products without affecting inclusion in Google Search. The source also notes that some media companies would likely prefer to fully opt out of AI features, while still remaining in Google Search.
That distinction is central. Training data controls are one thing; the use of content inside AI features connected to search is another. Prince’s argument is that publishers need a more meaningful choice than accepting broad Google access or risking the loss of important distribution.
The publisher dilemma
For media businesses, the decision is not abstract. Prince said that losing search means losing about 20% of your revenue. That makes blocking Google’s crawler difficult even for companies that object to AI use.
He also said the problem extends beyond search traffic. If a publisher blocks Google’s crawler, Prince said it blocks Google’s ad safety team, which can cause advertisements across platforms to stop working. In his view, that makes blocking the crawler a non-starter for many businesses.
The practical choice, as Cloudflare describes it, is therefore uneven:
- Allow Google’s crawler and remain available for search and advertising systems.
- Block Google’s crawler and risk major consequences for search-driven revenue and ad operations.
- Try to negotiate with other AI companies that do not have Google’s same search position.
Prince’s warning is that this setup could tilt the AI market toward Google before broader competition has a chance to develop. He said the result could be that the market has “effectively handed the game to Google.”
What Cloudflare wants instead
Prince’s proposed direction is a market with much broader competition. He described a scenario in which potentially thousands of AI companies compete to buy content from thousands of media businesses and millions of small businesses.
That model depends on separation. If Google’s search crawler and AI access are treated as inseparable, then many site owners may feel forced to accept AI use because they cannot afford the search or advertising consequences of blocking Googlebot.
Cloudflare has also provided the CMA with data about how Google’s crawler works. According to the source, the company says that data helps show why it is nearly impossible for other players to replicate the same success Google could have.
Prince described the CMA’s move to flag Google as a potential regulatory target as thoughtful. In his view, the regulator appears to understand that Google’s position is different from that of other AI companies because search is already embedded in how websites reach audiences.
Why the pressure is widening
Cloudflare is not alone in making this argument. Last month, Neil Vogel, CEO of People, Inc., made a similar point. People, Inc. is described in the source as the largest digital and print publisher in the United States and operates over 40 media brands.
Vogel called Google a “bad actor” and said media companies had no real choice but to let Google crawl their sites for AI content because of the way the crawlers were combined. His company had adopted Cloudflare’s solution to block AI crawlers that do not pay.
Vogel also said that system was working, with deal discussions underway with several large LLM providers. That detail matters because it shows the dispute is not only about blocking bots. It is also about whether publishers can create a paid market for the use of their content in AI systems.
The broader question for the CMA is whether Google’s search position should carry over into AI products without additional limits. Cloudflare’s answer is no: if AI companies are going to compete for web content, Prince argues Google should not be able to use search dependency as a built-in advantage.