A standardized approach to AI training permissions is taking shape as four major technology firms pledge support for a shared opt-out mechanism. Anthropic brought the proposal to the IETF, and by Friday, Google, OpenAI and Microsoft had all announced their backing. The initiative addresses a growing frustration among content creators who currently must craft individual rules for each company's web crawler.
The mechanism operates through robots.txt, the existing file that websites use to communicate with web crawlers. Publishers can now add a group containing a new "policy token" called __usage_ai-training and specify which content paths should remain off-limits for training purposes. Rather than managing multiple opt-out systems, a single rule can now apply across multiple AI companies simultaneously.
Kevin Kelley of Anthropic authored the IETF draft and announced it on the organization's AI preferences mailing list. Anthropic has already begun implementation work, though Kelley acknowledged the rollout will require time to handle unforeseen complications. "It'll take us some time to add support across our services and work through the inevitable complications and edge cases that I'm sure we'll discover," Kelley wrote.
Implementation Timeline
OpenAI intends to deploy the signal within the coming months, with its existing crawler controls remaining in effect. Google has set a target of early 2027 for support, according to its crawler documentation. Microsoft will activate the new preferences starting 1 February 2027, replacing the NoArchive and NoCache tags it has employed since 2023.
Microsoft is simultaneously introducing a separate page-level setting for grounding, the practice of anchoring AI responses to current web content. One configuration permits Microsoft's own applications like Copilot while blocking the grounding services the company provides to external clients.
Krishna Madhavan, principal product manager for Microsoft Bing, framed the initiative in terms of publisher autonomy: "Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work."
What the Standard Does Not Cover
The IETF draft characterizes the token as expressing a preference rather than issuing a binding directive about crawling behavior. Websites without such a group have made no preference statement, and existing crawler rules do not automatically transfer to this new system. Importantly, private agreements between individual publishers and AI companies can supersede the shared signal.
Microsoft clarified that the new preferences "do not establish licensing or payment arrangements." The company also warned that blocking AI grounding "will significantly reduce your content's visibility in AI-generated experiences," though it will not affect a page's ranking in Bing search results.
The draft remains an individual submission without formal IETF endorsement or standing in the standards process. Meta communications head Andy Stone expressed support for a unified signal on X, according to The Verge, though the company has not committed to implementation.
European Context
Google positioned the proposal as aligned with the EU's Code of Practice for general-purpose AI under the AI Act, which recognizes robots.txt as a primary mechanism for publishers to communicate machine-readable preferences.
The draft acknowledges contributions from representatives of the Financial Times, the BBC, Bertelsmann and the European Publishers Council in shaping the discussion.
Publisher demands for compensation remain a separate issue. Cloudflare instructed AI crawlers in July to either pay for access or face blocking, with its chief executive warning that bot traffic could eventually exceed human web traffic by a factor of 1,000. In Australia, the ABC and SBS have similarly called for AI companies to compensate them for news content.
Source: The Next Web
