Cloudflare is changing how website owners manage AI crawlers. Instead of forcing customers to use a broad all-or-nothing block, the company is giving them separate controls for different kinds of automated traffic.
The shift matters because not every bot does the same thing. A crawler that helps a page appear in search results is different from one that collects data for AI model training, and both are different from bots that act for individual users through tools like ChatGPT.
From One Switch To Three Choices
Since July 2024, Cloudflare customers have been able to block all AI crawlers with a single click. That gave site owners a simple defense, but it also treated very different forms of bot activity as one category.
The new system separates bot access into three categories: Search, Training and Agent. Search covers search engine indexing. Training covers data collection for AI model training. Agent covers bots acting on behalf of users, like ChatGPT.
That distinction gives site owners a more precise way to decide what kind of access they want to allow. A website may want search visibility while still rejecting crawlers that gather training data. Another site may be more open to user-directed agents but less comfortable with model training use.
The key change is control. Cloudflare is no longer framing AI bot management as a single yes-or-no decision. It is turning it into a set of purpose-based rules that better match the different ways automated systems use website content.
Ad-Supported Pages Get Stricter Defaults
Starting September 15, 2026, Cloudflare will block Training and Agent bots by default on pages that carry ads. The source explains the reasoning: advertising signals that a site wants human visitors.
Search crawlers will still be allowed under that default. That is an important distinction, because Cloudflare is not treating all automated access as harmful or unwanted. It is separating indexing from other uses of website content.
For site owners, the practical result is a default position that protects ad-supported pages from certain AI-related crawlers without cutting them off from search indexing. That default still leaves room for site owners to make their own choices through the new controls.
The policy also affects bots that serve more than one purpose. Multi-purpose crawlers that both support search and collect training data will be handled according to the strictest applicable rule. That gives bot operators a reason to make crawler purposes clearer instead of combining different functions under one identity.
Cloudflare Pushes For Purpose-Based Crawling
Cloudflare is pushing bot operators to split their crawlers by purpose. The company’s position is that site owners should not have to accept training or agent access just because they want to remain visible in search.
Cloudflare CEO Matthew Prince has previously criticized Google for bundling its search crawler with its AI crawler. The new controls appear designed to create pressure around that kind of bundling by making crawler purpose central to access decisions.
Prince also said in June 2026 that bot traffic had surpassed human traffic on the internet for the first time. According to the source, this was a milestone he originally did not expect until late 2027.
That context helps explain why Cloudflare is moving from a simple block to a more detailed framework. If automated traffic now represents a larger share of internet activity, website owners need tools that distinguish between useful, acceptable and unwanted bot behavior.
Enterprise Customers Get BotBase
Cloudflare is also launching BotBase for enterprise customers. BotBase is a searchable database of all known bots built into the Cloudflare dashboard.
The database shows how each bot is classified and how it uses content. It can indicate whether a bot only links to pages or reproduces them in full.
That extra visibility is aimed at customers that need more detailed oversight. Instead of managing bots only by name or broad category, enterprise users can inspect how a bot is understood by Cloudflare and what kind of content behavior is associated with it.
For organizations that manage large sites or sensitive content strategies, that kind of classification can shape access decisions. It gives teams more context before they allow or restrict a bot.
Verified No Longer Means Automatically Allowed
Cloudflare is also changing how it treats verified bots. Previously, all verified bots were allowed through automatically.
Going forward, category will determine whether a bot gets access. The label "Verified" will not be enough on its own.
Bot operators will need to show that they identify themselves honestly and do not abuse the access they receive. That change ties verification to behavior and purpose, not just identity.
The broader message is clear: Cloudflare wants bot access to be based on what a crawler does, not simply whether it is known. For site owners, the new tools create a more flexible way to handle AI crawlers while still preserving search access where they want it.
The new options are also available to users on the free plan. That means this shift is not limited to enterprise customers, even though BotBase itself is an enterprise feature.