AI Training Opt-Out: What the Shared Robots.txt Rule Means
Learn how the shared AI training opt-out works, when major platforms plan to support it, and what publishers should do about crawling, visibility and licensing.
One AI Training Opt-Out Could Simplify Publisher Controls
Publishers who want to keep their work out of AI training currently have to make a messy choice. They can maintain different rules for different companies - or accept that their content might be used under each service's policies anyway. A proposed shared signal in robots.txt would let publishers express that decision more consistently. Google, Microsoft, and OpenAI have said they support the approach, while Anthropic says it's working on implementation. The timing won't match perfectly across companies, and the idea isn't an approved internet standard yet.
The distinction matters. A training opt-out isn't the same thing as blocking crawlers, guaranteeing that content will never show up in an AI answer, or replacing licensing agreements. It's a machine-readable preference that participating companies say they'll respect. For publishers, the real question is how to use it without accidentally harming search discovery or other useful distribution. This article breaks down how the signal works, what the announced rollout plans look like, what it does (and doesn't) control, and a practical way to decide what to allow.
How should publishers prepare to use it?
The best way to prepare is to treat this as a content-governance decision, not just another line to add to robots.txt. Before changing anything, decide what you're trying to protect and what you still want people to discover. A news site, a software documentation business, and an independent creator can all reach different conclusions here, based on their revenue models and how they distribute content.
Start by mapping what you publish. Separate public editorial pages, subscriber-only material, press assets, documentation, user-generated content, and pages that contain personal or sensitive information. These categories can have different commercial and legal implications. A blanket rule is simpler, but a path-specific preference can preserve the uses that help your business while excluding material where training creates more risk than value.
Build a content and use matrix
For each content group, decide whether you want to allow or restrict these different activities:
- Search crawling and indexing: Can a search engine fetch and surface the page?
- Model training: Should the content be used to train or improve a generative model?
- Live grounding: Can an AI product retrieve the page at answer time and use it as a source?
- Commercial reuse: Is the content covered by a separate license or agreement?
Don't assume one setting solves all four. This token is specifically about the training preference. Microsoft has also described a separate page-level setting for grounding. That matters if you want, say, reporting excluded from training but still want a Copilot-style answer that cites a current article - or if you want to block commercial grounding while still allowing Microsoft's own products to access other content.
Audit existing rules before adding a new one
Before you add anything, inventory your current robots.txt directives and provider-specific controls. Google-Extended, GPTBot, and other crawler rules can still affect different products and different activities. The shared training preference is meant to complement some existing controls, not wipe them out. If your broad crawler rules conflict with a narrower training preference, you'll want to understand that interaction before relying on either one.
For example, a site might allow search crawling while marking a path with a training preference. But if an existing rule prevents a provider's crawler from accessing that path at all, the training preference may not be the only control in play. When provider documentation becomes available, test what actually happens. And keep a change log so your team can explain why a rule was added and what business goal it was meant to protect.
Assign an owner and review the outcome
This shouldn't be a single-team decision between technical people and "whatever the policy file says." Publishing ownership, search traffic, legal review, and platform operations should be involved. Assign one person to maintain the policy, but make sure the trade-offs are agreed across teams. Then review the configuration once provider implementations are available, and again after meaningful changes to your content or revenue model.
Track what matters to your business: organic search performance, referral traffic from AI products where you can measure it, licensing inquiries, and any changes in how content is attributed or used. Don't treat a short-term traffic dip or spike as proof the policy worked - or didn't. Search and product behavior change too. The goal is an informed decision based on evidence, not tuning a setting for its own sake.
What does the signal not control?
The biggest limitation is baked into the proposal itself: it's a preference about how a site wants its content used. It isn't an instruction about whether a crawler can fetch a page. If a site doesn't include this signal, it's not expressing a preference through this mechanism. And their existing crawler rules for different crawlers don't automatically function as an AI training opt-out.
So don't read "no training preference" as an opt-in. And don't assume a conventional crawl block equals the same thing as the proposed token. The draft is designed to make training preferences explicit. Its value depends on publishers adding the preference and services choosing to honor it.
It is not a licence or payment mechanism
Microsoft has said the preferences don't establish licensing or payment arrangements. That's a real limitation for publishers who want compensation, not just more control. A machine-readable opt-out can help draw a boundary, but it can't set a price, create a contract, or guarantee payment for access.
Private agreements can also override the public signal. If a publisher has negotiated specific terms with an AI company, those terms may define permitted uses separately from what appears in a public preference file. Legal and commercial teams should review how any agreements interact with technical controls, rather than assuming the configuration settles the question.
It does not guarantee visibility in AI answers
Training and live grounding are different uses. Training means using content in model development; grounding means retrieving pages during answer generation. A publisher can set preferences for each. Microsoft warns that blocking AI grounding will likely reduce visibility of content in AI-generated experiences, while saying its grounding preference won't change a page's position in Bing search results.
Google has also described a connection between Google-Extended settings and visibility in Gemini app answers. If a site uses the new token without specifying Google-Extended, Google says it will treat that as a block on Google-Extended. If a site does allow Google-Extended, Google says it may appear in Gemini app answers and drive clicks, according to Google Search Advocate John Mueller. That's product-specific behavior, not a universal promise of traffic.
So the trade-off isn't simply "protect content" versus "give it away." Restricting use may reduce how your work shows up in certain AI experiences. Allowing use may increase distribution, while leaving open questions about attribution, substitution, and compensation. Those outcomes vary by product and audience, so publishers shouldn't assume an AI referral is the same as a traditional search visit - or a paid license.
It does not bind every company
The announced supporters aren't the whole AI market. Meta communications head Andy Stone expressed support for a common signal, but didn't say whether Meta would honor this specific approach. Other companies may adopt different controls - or none at all. A single shared rule can reduce operational overhead when it's respected, but it can't replace provider-by-provider checks in a changing market.
The broader context may encourage adoption. Google says the proposal fits the EU Code of Practice for general-purpose AI under the AI Act, which recognizes robots.txt as a way for publishers to express machine-readable preferences. That alignment is relevant, but it doesn't turn an IETF draft into a universally enforceable standard, and it doesn't settle all the open questions around copyright and licensing.
How should businesses weigh visibility against control?
For most publishers, this isn't really a debate about whether AI use is good or bad in the abstract. The question is whether a specific use creates enough distribution, revenue, or user value to justify the loss of control over that content. Make the decision by product and content category, then revisit it as the market changes.
A practical framework comes down to three questions:
- What value does the content create when used? A continuously updated help center may benefit from being cited by assistants. Exclusive investigations or paid research may be more valuable when kept out of training or grounding.
- What value do you get back? Consider referral traffic, brand attribution, subscriptions, licensing revenue, and customer support deflection. Visibility alone isn't compensation, and an answer that mentions your work doesn't always send users to the original page.
- What does it cost to restrict access? Restricting use may reduce exposure in AI-generated experiences even if traditional search rankings stay the same. Estimate whether that matters to your audience and business model rather than assuming it won't.
Use different policies for different content
A publisher with a large archive may not need one rule for everything. Public product documentation might be suitable for grounding because accurate answers help users. Premium analysis might be excluded from training unless a commercial agreement exists. A press kit can still stay available so journalists and partners can find current company information.
This approach costs more operational effort than a blanket opt-out. You need clear content boundaries, reliable path management, and periodic testing. The upside is that the policy better reflects your actual priorities, instead of trying to solve multiple problems with one setting.
Treat the setting as part of distribution strategy
AI products are quickly becoming another place where people discover information, but the impact differs by sector. A local publisher may care whether an answer sends readers to a story. A SaaS company may value grounded support answers that resolve issues without generating support tickets. A research publisher may view access to its archive as a licensable asset and prefer a negotiated arrangement.
Cloudflare's July decision to offer a way to block AI crawlers unless publishers pay illustrates the commercial pressure behind these controls. In Australia, the ABC and SBS have also argued that AI companies should pay for news. Together, these positions show why a common opt-out is only one piece of the debate: publishers are also asking how content is licensed and how value should be shared.
Key takeaways for an implementation decision:
- Use the shared token to state a training preference - not as a substitute for access controls or contracts.
- Keep training distinct from crawling, indexing, and live grounding in your discussions.
- Don't retire provider-specific rules until each company documents how its implementation interacts with this token.
- Use path-specific rules when different content types have meaningfully different business value.
- Review outcomes using referral, attribution, and licensing evidence - not assumptions about AI visibility.
Conclusion: Treat the signal as one control, not a settlement
A shared AI training preference in robots.txt could reduce the burden of maintaining separate rules for every major AI company. Its real value will depend on consistent implementation, clear product documentation, and whether publishers can combine it with existing crawler and grounding controls without unintended side effects.
For now, map your content, separate training from crawling and live use, and decide which outcomes you care about. Keep existing controls in place while support is still rolling out. The signal can make a publisher's position easier to express - but it can't, on its own, settle questions about visibility, attribution, copyright, or payment.