Google and Bing Agree on One Robots.txt Rule to Opt Out of AI Training: Should Your Brand Block It?

Short answer: for most brands, no. On 30 October 2026, Google and Microsoft said they will honour a shared robots.txt signal that lets a website opt out of AI model training, and OpenAI has also committed to it, according to PPC Land. But blocking training is a different decision from blocking AI answers. If you sell to customers in Egypt, Saudi Arabia or the UAE and want your brand to show up when people ask ChatGPT, Copilot or Google's AI features, keep your marketing pages open and protect only the content you actually charge for.

The signal is not live yet. Microsoft says it will start honouring the new preferences on 1 February 2027, and Google says it aims to support the control in 2027. The rule itself is still an Internet-Draft (a published proposal) at the IETF (Internet Engineering Task Force), so the details can still change. That gives you a few months to make a calm decision instead of a nervous one.

What Google and Bing actually announced

The proposal grew out of the IETF's AI Preferences (AIPREF) working group. It defines a special name, `__usage_ai-training`, that you place on a User-Agent line in robots.txt. It is not a crawler. It is a label that tells participating AI companies whether you want your content used to train their models. The draft's own example looks like this:

• `User-Agent: __usage_ai-training`
• `Disallow: /` (no training on the whole site)
• `Allow: /press-kit/` (except this folder)

Google said the control "will help websites express their AI training preferences across multiple AI providers in a single step." It also said that if you opt out with the new token and do not mention Google-Extended separately, it plans to treat that as if you had disallowed Google-Extended, its existing AI control. Company-specific controls such as Google-Extended and OpenAI's GPTBot keep working alongside the shared signal.

Microsoft went further. Bing announced three separate settings: AI Training, AI Grounding (using your pages to build AI answers) and Limited Grounding (allowing Microsoft's own AI products but not third-party grounding services). Its timeline: temporary grounding controls in Bing Webmaster Tools from 15 December 2026 for sites that use the older NoArchive or NoCache directives, honouring the new preferences from 1 February 2027, a 1 March 2027 deadline for those sites to move to the new settings, and the temporary controls expiring on 30 September 2027.

Training and AI answers are two different decisions

This is the part most teams will get wrong. Training means a company uses your content to change what its model learns. Grounding means an AI assistant fetches your page when someone asks a question, often with a link back to you. Bing says it plainly: "You can block AI answers from using your content but doing so can reduce your visibility in AI experiences." It also says traditional Bing rankings are unaffected.

On Google's side, Google-Extended does not affect ranking or inclusion in Google Search, as Search Engine Roundtable reported when Google clarified the control. But Google also says websites that allow Google-Extended are eligible to appear in grounded responses in Gemini Apps. So a blanket opt-out can cost you citations in some AI products, even if your search rankings stay the same.

For a growth team, the real question is not "Do I like AI companies using my content?" It is "Where do my next customers ask their questions, and do I want my brand in that answer?" I explained why that matters in HubSpot Now Tracks Your Brand Inside ChatGPT Answers, which covers AEO (Answer Engine Optimization).

A simple decision rule: protect what you sell, open what sells you

• Block training on premium content: paid reports, course material, member-only research, proprietary data, anything that is the product itself.
• Keep open: your homepage, product and service pages, pricing explainers, blog posts, FAQs, case studies and press pages. These are the pages you want AI assistants to quote.
• Keep grounding allowed wherever you want to be cited. In Bing's new settings, that means leaving AI Grounding on even if you switch AI Training off.
• Do not add a broad block "to be safe". The draft says a wildcard (*) group does not count for the new token anyway, and a careless Disallow under a wildcard can stop normal search crawling.

Most startups and SMEs (small and medium-sized enterprises) in the region belong in the second group. Your blog is marketing, not a product. If a buyer in Riyadh asks an assistant which payroll software suits Saudi SMEs and your comparison page is blocked from AI answers, you have handed that answer to a competitor. Publishers and research firms whose content is the product face a different calculation, and for them the new token is good news.

What to do before February 2027

• Audit your current robots.txt. Check whether someone already added blocks for GPTBot, Google-Extended, ClaudeBot or CCBot, and whether that was a deliberate choice.
• Sort your content into two buckets: pages that sell you and content that is the product.
• Decide per bucket and write the decision down, so the next agency or developer does not quietly undo it.
• If you use NoArchive or NoCache today, check Bing Webmaster Tools after 15 December 2026 and set the new preferences before 1 March 2027.
• Recheck when Google publishes its final implementation in 2027, because the draft can still change.

This is also a good moment to make sure your analytics can see AI referrals at all. If you cannot measure traffic from AI assistants, you will not know what an opt-out costs you. I covered the basics in Your Marketing Tools Don't Talk to Each Other.

What is still uncertain

Three things to watch. First, the token sits in an individual Internet-Draft, not an approved standard, and PPC Land reports that some disputes in the working group are unresolved. Second, Google has only committed to 2027, with its crawler note pointing to early 2027. Third, the draft defines AI training only. It says nothing about search or real-time retrieval, so each company's own controls still decide whether you appear in AI answers.

FAQ

How do I stop AI from training on my website content?

From 2027, you will be able to add a User-Agent line for `__usage_ai-training` with a Disallow rule in robots.txt, and Google, Microsoft and OpenAI have said they will honour it. Until then, use each company's own control, such as Google-Extended for Google and GPTBot for OpenAI. Remember these are preferences that participating companies agree to respect, not a technical lock.

Does blocking AI training hurt my SEO?

Blocking training on its own should not change classic rankings. Bing says traditional rankings are unaffected, and Google-Extended does not affect Google Search. The risk is AI visibility: blocking AI grounding in Bing, or Google-Extended for Gemini Apps, can mean fewer citations in AI answers.

Should I block GPTBot and Google-Extended in robots.txt?

Only for content you sell or want to keep exclusive. For marketing pages, blogs and product pages, staying open usually helps your brand appear when people ask AI assistants about your category.

If you want a second pair of eyes on what to open, what to protect and how to measure your AI visibility, get in touch or see how I can help.

Sources

• Google and Bing back one robots.txt rule to opt out of AI training (PPC Land)
• Supporting a shared AI training control for websites (Google Search Central)
• Machine-Readable Training Preference Signal (IETF Internet-Draft)
• Google to support shared AI training control (Search Engine Roundtable)
• Microsoft Bing AI training and grounding controls (Search Engine Roundtable)
• Google-Extended Bot (Token) Does Not Affect Google Search (Search Engine Roundtable)