Google, Microsoft, OpenAI and Anthropic will honor a shared robots.txt signal that lets websites opt out of AI training

One rule for every provider

Google, Microsoft, OpenAI and Anthropic said on Oct. 30 that websites will be able to state, with one shared signal in their robots.txt file, that their content should not be used to train AI models. Until now, site owners had to write a separate rule for each company, such as Google-Extended for Google or GPTBot for OpenAI. Under the new approach, a site owner puts the __usage_ai-training token on the User-Agent line where a crawler name would normally go. The paths that should stay out of training are then listed with the usual Disallow and Allow rules.

In the draft’s example, a site writes Disallow: / and Allow: /press-kit/ under User-Agent: __usage_ai-training. That expresses a preference against training for everything except the /press-kit/ folder, and a preference in favor of training for that folder.

The signal is defined in “Machine-Readable Training Preference Signal” (draft-kelley-aipref-training-preference-signal-00), which Anthropic’s Kevin Kelley submitted to the IETF on Oct. 30. It is an individual Internet-Draft that the IETF has not endorsed; it is not a standard, and it expires on May 3, 2027. The draft adds no new directives or grammar. It keeps the robots.txt language of RFC 9309, its longest-match precedence and its group structure as they are. It takes its definition of AI training from the vocabulary draft of the IETF’s AIPREF working group: using an asset to modify the learned parameters of a generative AI model.

What it covers and what it does not

According to the draft, the token is a usage preference, not a directive for crawling. Crawlers keep applying the rules written for their own product tokens. A site owner who also wants to limit crawling has to do that separately, with groups for those crawlers. Wildcard groups (User-Agent: *) must not be read as matching the shared token, so a site that blocks all bots has not expressed an AI training preference. The draft says specific expressions should take precedence over general ones and accepts that private agreements may override these preferences.

The definition covers training only. The draft does not address grounding, where web content is pulled in as a source while an answer is generated, or search. According to AFP, Microsoft said the setting applies only to future model training and does not stop bots from crawling sites. Content stays available by default until a site owner changes the setting. Google and Microsoft each said in separate posts that opting out could reduce the chances of content surfacing in AI apps and experiences.

The companies’ timelines

According to a post by Krishna Madhavan, a product manager at Bing, Microsoft will honor the new preferences as it crawls starting Feb. 1, 2027. Microsoft separates training from grounding: both can be set separately at page level with a meta tag or an X-Robots-Tag header. Grounding has three values. Allow is the default; Disallow keeps content out of generated responses; DisallowExtended limits use to Microsoft-operated experiences such as Copilot and Microsoft 365. As a transition, temporary grounding preferences open in Bing Webmaster Tools on Dec. 15, 2026, for sites that use NoArchive or NoCache. After March 1, 2027, those two tags will no longer be read as training or grounding preferences, and the temporary preferences expire on Sept. 30, 2027. Microsoft says the preferences do not change eligibility for traditional Bing search results.

In a post on the Google Search Central blog, John Mueller wrote that Google aims to support the new control along with other AI providers in 2027; Google’s crawler documentation says “early 2027.” If a site opts out through the shared token and has no rule for Google-Extended, Google plans to treat that as if the site had disallowed Google-Extended. According to the documentation, Google-Extended covers both Gemini model training and grounding in Gemini apps, but it does not affect a site’s inclusion in Google Search and is not a ranking signal. PPC Land notes that because Google offers no separate grounding token, a site that opts out of training through the shared token also drops out of Gemini responses. Google says existing company-specific controls will keep working alongside the new signal and that the proposal is consistent with the EU’s Code of Practice for General Purpose AI model providers under the AI Act.

OpenAI aims to implement the signal in the coming months. According to PPC Land, the company said the new signal will not replace the controls it already provides through robots.txt. Anthropic has published no blog post or announcement on the matter; its only statement is Kelley’s message to the IETF’s ai-control mailing list. Kelley wrote that Anthropic has already started implementing support for the draft, that it complements the company’s existing crawler controls and that adding support across its services will take time.

Who has not committed

Meta is not among the four companies. According to AFP, Meta spokesperson Andy Stone wrote on X that creating a common signal that conveys such preferences to multiple providers at once is an important step, but Meta did not commit to honoring it. No statement on the signal was found from Amazon or Apple. According to PPC Land, the draft’s acknowledgments list people from Meta and AWS among participants in the AIPREF discussions, which is not a commitment to implement it.

AFP notes that publishers are dealing with a surge of bots collecting their content for training. The New York Times sued OpenAI and Microsoft in December 2023, alleging its articles were used without permission or payment; the two companies have invoked fair use. Cloudflare has blocked AI bots by default on new customers’ sites since July 2025.