Website owners may soon have a simpler way to say no to AI training. On Friday, 30 October 2026, Anthropic, OpenAI, Google and Microsoft said they will support a common robots.txt signal that lets sites state their AI training preferences once, instead of blocking each company’s crawler one by one.
What is changing?
Today, a site that does not want its content used to train AI models has to add separate rules for each company’s crawler, and keep that list up to date as new bots appear. The new signal is designed to express one preference that multiple participating companies will respect. OpenAI says the goal is to make it easier for publishers and site operators to communicate the same preference to several providers at once.
Who proposed it?
The proposal comes from Kevin Kelley, an Anthropic engineer, as an Internet Engineering Task Force (IETF) draft on a training-preference signal. It is still a draft, so details can change before it becomes a standard.
When will each company support it?
- Anthropic: already supporting it
- OpenAI: says it will support it in the coming months
- Google and Microsoft: say they will support it next year
- Meta: a spokesperson called it an important step and said the company supports publishers’ ability to express preferences about generative AI training
Because the proposal comes from an AI company, it is worth reading the IETF draft yourself and judging it on its merits.
Meta pauses its scraping terms
Separately, Meta has been under scrutiny for proposed commercial terms that legal experts say would allow broad scraping of news sites. Last week it agreed to delay putting those terms into effect.
What should site owners do now?
- Keep your existing rules: if you already block specific AI crawlers in robots.txt, leave those rules in place. The new signal is not supported by every company yet.
- Read the IETF draft: check the exact wording before you add anything to your robots.txt, since the syntax could still change.
- Decide your policy: choose whether you want to allow or opt out of AI training and whether that applies to your whole site or only some sections.
- Do not confuse it with search: blocking AI training is separate from allowing search crawlers such as Googlebot, so make sure you do not accidentally block your search visibility.
- Review regularly: robots.txt is a request that well-behaved bots follow, not a technical block, so use other protections for sensitive content.
Details are based on reports published on 2 November 2026. The standard is a draft and company timelines may change.
