Cloudflare is moving toward a new bargain between websites and AI companies: if bots want to use a publisher's content, the site owner should be able to see it, control it, and potentially charge for it.
The company announced plans on Monday to launch a marketplace in the next year where website owners can sell AI model providers access to scrape their sites. Before that marketplace arrives, Cloudflare has launched AI Audit, a set of free observability tools designed to show customers when AI crawlers arrive, where they come from, and how often they request content.
Why Cloudflare Is Targeting AI Scraping
The issue Cloudflare is trying to address is simple but consequential. AI model providers scrape thousands of small websites for information that helps power their LLMs. Larger publishers have been able to negotiate licensing deals with OpenAI, but most websites receive nothing while their work is still used by popular AI models on a daily basis.
Cloudflare CEO Matthew Prince framed the problem as a direct threat to online publishing. In an interview with TechCrunch, he said: "If you don't compensate creators one way or another, then they stop creating, and that's the bit which has to get solved,"
That concern is not only about principle. If users get answers from ChatGPT instead of visiting the original website, smaller publishers may lose traffic they rely on. For sites whose business models depend on visitors, advertising, subscriptions, or other forms of direct audience relationship, that shift can weaken the reason to keep producing original material.
What AI Audit Gives Website Owners
Cloudflare's first step is visibility. AI Audit gives website owners a dashboard showing analytics about why, when, and how often AI models crawl their sites for information. The tool is meant to make bot activity easier to inspect instead of leaving site owners to guess what is happening in server traffic.
A demo shared with TechCrunch showed that AI Audit can identify where each scraper visiting a site comes from. It also offers selective windows into how many times scrapers from OpenAI, Meta, Amazon, and other AI model providers are visiting.
The second step is control. Cloudflare customers can block AI bots from their sites with one click. They can also block all web scrapers through AI Audit, or allow certain scrapers through when they have a deal in place or believe the scraping is useful.
That selective approach matters because not every publisher wants the same rule for every bot. Prince told TechCrunch that customers are asking for tools that let them choose which AI models can access their sites. A publisher may want to block one crawler while allowing another, especially when business arrangements or perceived benefits differ.
From Blocking Bots to Setting Terms
Cloudflare had already released a one-click button to block all AI bots shortly after AI-powered search startup Perplexity was accused earlier this summer of scraping websites that had indicated they did not want to be crawled using the Robots Exclusion Protocol.
Prince described the earlier move as a response to publisher frustration. He said: "That was out of frustration we were hearing, where people were feeling like their content was being stolen,"
The pressure is also technical. Some website owners told Business Insider that AI bots were scraping their websites so heavily that it felt like a DDoS attack was affecting their servers. Cloudflare's argument is that scraping can create real costs for website operators because it can raise cloud bills and affect service.
AI Audit therefore sits between two needs. It gives publishers defensive controls today, while preparing the ground for a marketplace where access may become something they can price, approve, or exchange for credit.
How the Marketplace Could Work
Cloudflare says the marketplace, expected sometime in the next year, is intended to help smaller publishers do something that has mostly been available to bigger platforms and publishers. Prince described the idea this way: "Let's give all of you the ability to do what only Reddit, Quora, and the big publishers of the world have done previously,"
He added: "What if we let you set, effectively, a price for accessing and taking your content to ingest into these systems."
The basic proposal is that websites could charge AI model providers based on how heavily they scrape individual sites. Cloudflare has not provided a complete picture of how the marketplace will operate, and Prince said the terms could vary. A site might charge money for scraping, or it might ask an AI lab to provide credit instead.
That lack of detail is important. The marketplace is a bold concept, but Cloudflare has not yet explained how much AI companies will actually pay, how publishers will set prices, or what the final product experience will look like. The company is outlining a direction rather than publishing a finished rulebook.
Why Better Measurement Matters
The same visibility problem affects large publishers, too. Prince said that even major publishers with OpenAI licensing deals, including TIME, Condé Nast, and The Atlantic, have relatively little insight into how much ChatGPT is scraping their websites.
According to Prince, many of those publishers have to accept what OpenAI tells them. That answer matters because it helps determine whether the licensing deal is good for the publisher.
For smaller websites, the stakes are sharper because they may not have licensing agreements at all. Cloudflare is betting that a combination of analytics, blocking controls, and a future marketplace can give those publishers more leverage.
AI companies may not be eager to pay for content they currently scrape without payment. But Prince argues that the current arrangement, where some AI companies never pay for content, cannot continue indefinitely. Cloudflare's plan is to make scraping visible first, negotiable next, and potentially paid after that.