Cloudflare is testing a new answer to one of the web’s biggest AI-era problems: what should happen when AI companies crawl publisher websites but do not send much reader traffic back?
The cloud infrastructure provider, which serves 20% of the web, announced Tuesday that it is launching Pay per Crawl, a marketplace designed to let website owners put a price on AI bot access. The product is starting as an "experiment" in private beta, but the idea is much larger: make crawling permission-based, measurable, and potentially paid.
How Pay per Crawl works
Pay per Crawl gives website owners three basic choices for each AI crawler. They can charge a set rate, allow scraping for free, or block access entirely. The paid option works as a micropayment for every single "crawl."
Cloudflare says its tools can also show website owners why crawlers are visiting. According to the company, publishers can see whether a crawler is collecting material for AI training data, for inclusion in AI search responses, or for another purpose.
Both sides need Cloudflare accounts to take part in the experimental marketplace. Publishers can set the rate at which they want to sell access to a "crawl," while AI companies can set the rates at which they want to buy that access. Cloudflare sits between the two, charging the AI company and passing earnings to the publisher.
That intermediary role matters. Cloudflare is not only offering a publisher tool; it is positioning itself as the infrastructure layer for a possible new content economy between websites and AI systems.
Why publishers are looking for new leverage
The launch arrives as news publishers face difficult questions about how readers will find their work. The old arrangement with search was imperfect but clear: publishers let Google crawl pages, Google Search sent users back, and those visits could turn into ad revenue.
Cloudflare argues that the AI era may offer a much weaker exchange. This June, Cloudflare says Google’s crawler scraped its websites 14 times for every referral it gave them. OpenAI’s crawler scraped websites 1,700 times for every one referral, while Anthropic scraped websites 73,000 times for every referral.
Those ratios point to the central tension. Crawling may help AI products answer user questions, but if the answer appears inside a chatbot or AI search response, the publisher may not get the same visit it once expected from a search results page.
Some websites cite ChatGPT as a major traffic source, but Cloudflare says that does not appear to be broadly true. For publishers that depend on readers reaching their own sites, that difference is not a technical detail. It affects the business model.
Default blocking changes the starting point
Cloudflare also announced Tuesday that new websites created with Cloudflare will now block all AI crawlers by default. Site owners will have to approve specific AI crawlers before those crawlers can access their sites.
Cloudflare says the change gives every new domain "the default of control." In practice, it shifts the starting position from open access to permission first. That aligns with the company’s broader push for a "permission-based approach to crawling."
Several major publishers have signed on with Cloudflare to block AI crawlers by default in support of that goal. The list includes Conde Nast, TIME, The Associated Press, The Atlantic, ADWEEK, and Fortune.
The company has been preparing this direction for more than a year. It previously launched tools including a one-click way to block all AI bots and a dashboard showing how AI crawlers visit a site. In a 2024 interview with TechCrunch, Cloudflare CEO Matthew Prince said those products were building toward a marketplace where publishers could distribute content to AI companies and receive compensation.
The licensing gap Pay per Crawl is trying to fill
Publishers have already tried different approaches to AI companies. Some, including The New York Times, have filed lawsuits against tech companies over training AI models on news articles without permission. Others have signed multi-year licensing deals for AI model training and for appearing in AI chatbot responses.
But the source article notes a major limitation: only large publishers have reached those AI licensing deals. It is also unclear whether those agreements create meaningful revenue. Cloudflare’s pitch is that a broader marketplace could let more publishers set prices on their own terms.
Pay per Crawl does not guarantee that publishers will get attractive economics. It also depends on AI companies deciding to participate, even though they can currently scrape content for free in many cases. That is a significant adoption challenge for the model.
Still, Cloudflare has unusual reach because it already sits in front of a large share of the web. If enough publishers and AI companies join, the marketplace could turn crawling from a hidden background activity into a visible transaction.
What an agentic web could change
Cloudflare frames Pay per Crawl as especially relevant to a future shaped by AI agents. OpenAI and Google are building AI agents intended to visit websites for users, gather information, and bring it back directly. If that becomes common, publishers that rely on site visits could face even more pressure.
Cloudflare says the "true potential" of Pay per Crawl may appear in an "agentic" future. In a blog post, the company asked: "What if an agentic paywall could operate at the network edge, entirely programmatically? Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho — and then giving that agent a budget to spend to acquire the best and most relevant content,"
For now, Pay per Crawl does not use stablecoins or cryptocurrency, according to Cloudflare spokesperson Ripley Park. But during a Tuesday appearance on the TBPN show, Prince said Cloudflare was looking into "potentially creating our own stablecoin that would be part of how these transactions take place," as well as working with other stablecoin providers.
The marketplace is still an experiment, and its success is not assured. But it gives publishers a clearer set of options than simply accepting crawler traffic or trying to block it after the fact. In the AI search era, that control may become one of the web’s most important bargaining tools.