Four AI Giants Unite Behind a Single robots.txt Opt-Out Signal

Publishers who have spent the past two years juggling a separate blocking rule for every AI crawler finally have a simpler option on the table. Google, Microsoft, OpenAI and Anthropic have committed to honoring a single shared signal in robots.txt that lets a website refuse the use of its content for training generative AI models. The mechanism comes from an Internet-Draft authored by Anthropic’s Kevin Kelley and published on October 30, 2026, with all four companies announcing support the same day.

• One token replaces many: Sites write a new group keyed to __usage_ai-training instead of naming each company’s crawler individually.
• Timelines vary widely: Anthropic has already begun work, OpenAI targets the coming months, Google aims for early 2027 and Microsoft commits to February 1, 2027.
• Voluntary and training-only: The signal carries no legal force, covers training rather than crawling, and does not establish any licensing or payment arrangement.

How the shared signal actually works

The design deliberately reuses the syntax webmasters already know. Rather than a bot name, the User-Agent line carries a policy token, and the usual Allow and Disallow rules follow beneath it. The draft’s own example blocks an entire site while carving out a press section:

User-Agent: __usage_ai-trainingDisallow: /Allow: /press-kit/

Because no new grammar or directive names are introduced, path matching and longest-match precedence follow RFC 9309, the existing robots.txt specification. Parsers that do not recognize the token simply ignore the group. The draft ties the token to the AIPREF working group’s definition of AI training: using an asset to modify the learned parameters of a generative model. Models built exclusively for non-generative tasks fall outside that scope.

Several nuances matter. A site without the group has expressed no preference at all, and wildcard rules aimed at all crawlers do not count as an opt-out. Private agreements between a publisher and an AI company can override whatever the file says. Most importantly, the token states a preference about content use, not an order about crawling; blocking visits still requires naming each crawler.

Company-by-company rollout and the grounding trade-off

Anthropic has already started implementing support, according to Kelley, who described the work as complementing existing crawler controls rather than replacing them and warned that covering every service will take time. OpenAI said it aims to ship the signal in the coming months while keeping controls such as GPTBot in force. Google, per Search Advocate John Mueller, targets early 2027 and will treat an opt-out that omits Google-Extended as a block on Google-Extended itself. Microsoft is the most specific: it will honor the preferences from February 1, 2027, and from March 1, 2027 will stop reading its NoArchive and NoCache tags as training or AI-answer preferences.

Microsoft also separates training from grounding, the use of live pages to feed generated answers. Its page-level setting offers three choices: allow, disallow, or an intermediate option that permits use in Microsoft products such as Copilot while blocking the grounding services it sells to other companies. Krishna Madhavan, principal product manager for Microsoft Bing, framed the rationale around publisher choice.

Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work.

Microsoft cautions that blocking grounding significantly reduces a brand’s visibility in AI-generated experiences without changing a page’s position in ordinary Bing results. Google makes a parallel point: sites that permit Google-Extended remain eligible to appear in Gemini app answers and receive clicks.

Europe, Meta’s silence and the limits of goodwill

Google linked the proposal to the EU Code of Practice for general-purpose AI under the AI Act, which recognizes robots.txt as a primary route for machine-readable publisher preferences. The draft thanks staff from the Financial Times, the BBC, Bertelsmann and the European Publishers Council for shaping the discussion. Meta is the notable holdout: communications head Andy Stone voiced support for a common signal on X but did not say whether the company would honor it.

The status of the document itself deserves scrutiny. It is an individual IETF submission, not endorsed by the organization and carrying no formal standing in the standards process, and it expires on May 3, 2027 unless revised. Compliance rests entirely on corporate goodwill, with no audit mechanism and no penalty for ignoring the token. Nor does it touch the money question that news organizations keep raising, and it cannot undo scraping that has already happened.

For all those caveats, consolidating four separate configurations into one written definition of training is a genuine structural improvement for anyone publishing on the open web. The real test arrives in 2027, when the announced dates pass and the industry learns whether a voluntary preference behaves like a standard or merely like a press release.