Opting out of AI training will soon take a single line in your site's robots.txt

Until now, a website publisher who didn't want their texts used to train artificial intelligence had to block each AI crawler one by one. One line for OpenAI's bot, another for Google-Extended, a third for Anthropic, and so on, as new crawlers appeared. On October 30, Google, Microsoft, OpenAI and Anthropic announced that they would honor one and the same directive.

It goes in the robots.txt file, the small file that has told crawlers what they may visit for more than thirty years. The signal has a technical name, __usage_ai-training. It is described in a draft filed on October 30 with the IETF, the body that develops Internet standards. Its author, Kevin Kelley, works at Anthropic.

How to write it in your robots.txt

The signal is used like a crawler name. The draft gives this example, which refuses training on the whole site except the press kit folder:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

One detail matters a lot. According to the draft, a general User-Agent: * rule must not be interpreted as targeting this signal. A site that already blocks all crawlers therefore expresses no preference on training until it adds this group. Without it, no choice is expressed.

Who is committing, and starting when?

The timelines differ. Microsoft is the most specific: its Bing search engine will start reading these preferences on February 1, 2027, and gives sites until March 1, 2027 to publish them. After that date, its older NoArchive and NoCache tags will no longer count as a refusal of training. Microsoft is adding a page-by-page setting and a second signal to refuse that its AI use a piece of content to build its answers.

Google plans to support the signal in 2027. It will treat it as an opt-out addressed to Google-Extended, its existing setting, according to trade press that read its announcement. OpenAI speaks of “the coming months,” according to accounts of its blog post. At Anthropic, the draft's author wrote on an IETF mailing list that the company had “already started” implementing it.

Meta, which recently extended its AI training to its advertisers' sites, does not appear among the companies making a public commitment.

What this signal does not do

The text is explicit: it expresses a preference, not an access ban. Crawlers can keep visiting your pages. Nothing removes your content from models that have already been trained. The draft has no official status at the IETF, and private agreements may override the stated preferences.

Microsoft also points out that opting out guarantees nothing in return, neither traffic nor payment, and neither does opting in. In Europe, Google presents this signal as compatible with the AI Act's code of practice for general-purpose AI; that is its own assessment, not Brussels'.

What to do right now

There is no rush before 2027, but the decision deserves to be made early, with management and the legal department:

• take stock of the current rules in your robots.txt that target AI crawlers (GPTBot, Google-Extended, ClaudeBot…) and keep them: they remain valid;
• decide which parts of the site can be used for training (press releases, public documentation) and which cannot;
• if Bing matters to you, note the March 1, 2027 deadline.

It remains to be seen whether the draft will clear the IETF's stages, and whether other players, in China as in Europe, will rally to it.