Publishers may soon be able to express AI scraping preferences in one place. OpenAI, Google, Microsoft, and Anthropic say they will adopt a shared signal intended to let websites tell AI crawlers whether their content can be used.
One signal for multiple crawlers
The proposed “policy token” would sit in a website’s robots.txt file. That file already communicates which crawlers may access a site for purposes such as search indexing, archiving, or AI training.
Today, publishers need to communicate their AI training preferences separately to each AI company. A shared token could reduce that repeated work by giving participating crawlers the same signal to read. The approach is being proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF).
The change is aimed at making publisher preferences easier to state across services. Its practical effect depends on AI companies recognizing the signal and restricting their crawlers as they have said they will.
What the companies say they will do
OpenAI, Google, Microsoft, and Anthropic say their AI crawlers will be restricted from accessing a site when they encounter the shared token. Their stated implementation plans differ: Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the “coming months.”
Microsoft says it will also offer more specific controls. Publishers will be able to opt out of AI training or AI grounding, which is the process of directing an AI model to a verified information source to reduce hallucinations.
Microsoft also says a website will continue to appear in Bing search results regardless of the publisher’s preferences. That distinction matters for publishers deciding how to communicate restrictions: the stated choices affect AI uses, while search visibility is described separately.
A response to growing scraping concerns
The proposal comes as publishers deal with a continuing influx of AI crawlers accessing websites and feeding content into AI models. The shared token is one of several approaches mentioned in the effort to give publishers more control over this activity.
Another is the Really Simple Licensing (RSL) standard, which gives publishers the ability to charge crawlers that visit their sites. Cloudflare has also begun blocking AI crawlers by default. These approaches take different routes: one sets a shared preference, one enables charging, and one blocks crawlers by default.
The shared signal’s appeal is its potential to apply a preference across multiple companies without publishers having to configure it separately for each one. But the announcement describes a commitment from the named companies to restrict their crawlers when they see the token; it does not describe a universal technical enforcement mechanism for every crawler on the web.
What publishers should watch
For publishers, the key details are whether each company implements the signal as promised and how its controls distinguish among uses such as training and grounding. Microsoft’s more granular options suggest that a single broad preference may not cover every decision a publisher wants to make.
The timelines also mean support will arrive at different points. Anthropic’s adoption is already in place, while Google and Microsoft have pointed to next year and OpenAI to the coming months. Until support is implemented across these services, the shared token’s value will depend on which crawlers recognize it.
The proposal offers publishers a common way to communicate AI preferences through robots.txt. It could make those preferences simpler to manage across participating companies, while leaving publishers to consider how the signal, company-specific controls, and other scraping responses fit together.