Google, Anthropic, OpenAI and Microsoft have committed to honoring a new robots.txt directive allowing publishers to specify preferences for content scraping for generative AI training. The initiative, written by Anthropic technical lead Kevin Kelley and under consideration at IETF, standardizes how AI companies interpret publisher preferences. Currently, website owners must identify individual crawlers and add separate rules for each provider.
The new proposal introduces a single common signal, requiring publishers to set up a new user agent in robots.txt: User-Agent: __usage_ai-training Disallow: /Allow: /press-kit/. This blocks crawlers meant to gather data for AI training while allowing specific sections like press kits to remain accessible. The directive does not prevent general web crawling or content from appearing in search results.
Google will interpret it as a stop sign for the Google-Extended agent training Gemini models. Anthropic has already implemented support, Microsoft will follow on February 1, 2027, Google in early 2027, and OpenAI in the coming months. The measure cannot reverse past scraping already performed by these companies.
Source: Engadget · Summarized by HeadlinesBriefing