AI Scraping Opt-Out: What Publishers Need to Know

What is the shared AI scraping opt-out?

The proposal is a shared "policy token" signal that publishers can use to express preferences about AI use of their content. It's meant to live in robots.txt, a file websites have long used to give crawler instructions. Search engines, archives and other automated services often check that file when deciding what to crawl.

The main difference is coordination. Today, publishers may need to communicate AI preferences using different crawler names, separate rules, or company-specific controls. A shared signal would let multiple participating providers treat one instruction the same way. That can cut down on how many separate knobs a publisher has to manage - assuming crawler operators actually implement it as intended.

The signal was proposed by Kevin Kelley, a technical lead for standards at Anthropic, through work associated with the Internet Engineering Task Force (IETF). After the announcement, Anthropic said it had already adopted it. Google and Microsoft said they planned to support it the following year, while OpenAI said implementation was expected in the coming months. Those were plans - not a guarantee of the same timeline or the same behavior across providers.

One practical point matters a lot: robots.txt is a machine-readable request, not an access-control system. A compliant crawler can read it and choose whether to follow it. The file doesn't stop someone from requesting public pages, and it can't delete or undo material already collected.

For publishers, the proposal is best thought of as a shared protocol for expressing intent. Its value depends on two things: whether providers interpret it consistently, and whether they actually honor it in their systems. Standardization can reduce operational friction, but it can't settle the legal or commercial question of whether a specific use of content is permitted.

Why do publishers need a common signal?

AI crawlers create a coordination problem. Publishers want control over how their pages are used, while AI companies need large amounts of material for different products and workflows. In practice, current controls often shift the administrative burden onto the publisher: figure out which crawlers matter, understand what they're doing, keep the rules current, and still confirm whether the provider recognizes them.

That burden gets worse as organizations run more domains, subdomains, or content systems. For example, a media group might have a news site, a product-review site, and a separate archive. Each property can have different business requirements, but the same team still has to maintain crawler rules across all of them. A common signal won't eliminate governance work - but it can make the basic instruction easier to apply.

This proposal also fits into broader efforts to manage automated access. Some publishers have looked at licensing approaches, including the Really Simple Licensing standard, and Cloudflare has offered ways to block AI crawlers by default. Those approaches address different questions: licensing is about terms and value exchange; network-level blocking is about access at the infrastructure layer; a shared robots.txt signal is about expressing a preference for how crawlers should behave.

Those distinctions matter because publishers don't all want the same outcome. One company might want its content excluded from model training while still allowing it to show up in search. Another might allow retrieval of pages to answer a user question, as long as the source is identified, while rejecting use of the same material for training. Still another might block most automated access unless it's explicitly approved. A common standard helps with consistent communication across providers, but it won't act like a single "switch" that resolves everything about attribution, compensation, data retention, or downstream use.

What uses of content can publishers control?

The announcement draws an important line between AI training and AI grounding. Training uses data to develop or update a model. Grounding - often described as retrieval-augmented generation - directs a model to relevant sources when it's generating an answer. Using sources at response time isn't the same as training on them.

Publishers may want different rules for each. A financial publisher might not want its reporting included in training data, but still want an AI assistant to retrieve a current article and cite it when answering users. A reference publisher might allow indexing for discovery while limiting model development use. Treating "AI access" as one category can blur those differences.

Microsoft said it would offer more granular controls for opting out of training or grounding. It also stated that websites would continue to appear in Bing search results regardless of those preferences. Google has previously offered similar controls for whether content is used in AI search features or for training, without affecting ordinary search visibility. The takeaway is simple: check what each setting actually changes, rather than assuming that "opt out" means the same thing everywhere.

Before you change crawler rules, write down your intended outcome in plain language. For example:

• Should the content be excluded from model training?
• Should it remain available for search indexing?
• Can an AI system retrieve it to answer a user, as long as the source is identified?
• Does the policy apply to the whole site, or only certain sections and content types?
• Are there contractual, privacy, or licensing obligations that affect your decision?

These questions aren't just technical. Blocking access may reduce the chance your content shows up in some AI-generated answers, but it could also affect discovery or referral traffic depending on the provider and the use case. Allowing access might preserve visibility, while still leaving open questions about attribution and value exchange.

In the end, the "right" policy depends on the business model. A subscription-based publisher may prioritize protecting exclusive material. A public-service organization may prioritize broad access and accurate representation. A specialist publisher might allow some uses while negotiating terms for others. The standard can help you communicate a choice - it can't make that choice for you.

How should a publisher prepare to use the standard?

Don't start by editing robots.txt in production. Start by defining the policy, assigning ownership, and understanding how crawler instructions are managed today. Even a technically valid rule can create operational trouble if it blocks a useful service, conflicts with an existing policy, or gets copied inconsistently across domains.

A practical preparation process looks like this:

1. Inventory the properties you control. List domains, subdomains, regional sites and content platforms. Confirm which teams can change crawler policies.
2. Separate content uses. Decide whether training, grounding, search indexing and other crawling should have different rules. Avoid a blanket position if each use case needs different treatment.
3. Review existing directives. Check current robots.txt rules and any provider-specific controls. Document what each rule is intended to do and whether it still matches the organization's policy.
4. Track the standard's implementation. Confirm the final syntax, scope and behavior from the standard-setting work and from each provider's documentation before deploying a token. The announcement alone isn't a full implementation guide.
5. Test and monitor. Validate that the file is served correctly, check that it's consistent, and monitor crawler activity where possible. Revisit the policy if providers change their tooling or if the business model changes.

Ownership is where a lot of teams get tripped up. You might have an editorial team that sets policy, a legal team that reviews rights questions, and an engineering team that actually deploys the rules. If nobody owns the whole workflow, a decision made in one department can get undone by a website migration or a new content platform.

Keep a short decision record next to the technical configuration. Include the goal, what content is covered, the date of the review, and who is responsible. When outcomes change - like a provider's traffic dropping - you'll be able to connect it to a known decision instead of guessing which configuration caused the shift.

Also, avoid treating crawler logs as proof of compliance. Logs can show that a known crawler requested a page, but they can't tell you what happened after that or whether content was later stored or used. They're an operational signal, not an audit trail of training data.

What can the opt-out not guarantee?

The biggest limitation is enforcement. robots.txt communicates a preference to crawlers that choose to respect it; it doesn't technically stop access. A provider's promise to reduce or restrict crawling after seeing the signal gives the mechanism real-world value, but publishers still rely on providers to implement and honor it.

Coverage is another limitation. The announced support includes OpenAI, Google, Microsoft and Anthropic, with Meta saying it supports a shared signal but without confirming it will honor the setting. Other crawlers may not recognize it at all. And providers may run multiple crawlers for different products - so verify which systems a rule covers instead of assuming it's universal just because the company name is familiar.

There's also a timing issue. A new setting affects future behavior after it's implemented and detected. It doesn't automatically delete content already collected, remove material from existing datasets, or guarantee that a model "forgets" what it learned previously. Those outcomes require separate steps and can't be inferred from a crawler preference alone.

Finally, technical signaling and legal rights aren't interchangeable. The standard can make a publisher's preference easier for providers to read consistently, but it doesn't determine whether past or future use is lawful, create a payment obligation, or guarantee attribution. Those questions depend on jurisdiction, contract, and the specific use case. If you deal with high-value or sensitive content, it's wise to involve legal review alongside technical controls.