Today, OpenAI announced that it will honor a common signal in robots.txt, the file that lets websites express their preferences for the use of their content in foundational AI training. This voluntary effort simplifies communication for publishers and website operators, who previously had to create separate rules for each provider’s crawler. By adopting a shared signal, operators can convey the same preference to multiple participating services at once, reducing the workload of maintaining individualized crawler rules.
The new signal does not replace existing robots.txt controls, which continue to provide publishers with flexibility and control over their pages. OpenAI plans to implement the signal in the coming months and will issue another update with full implementation details when ready. The initiative builds on the ongoing work of the IET F AI Preferences working group, of which OpenAI is an active participant, and reinforces its commitment to open‑web standards and fairness.
We believe this step supports an open web where information can be published and discovered while giving site owners clearer, simpler ways to communicate their content‑use choices. OpenAI encourages other AI providers to adopt the signal as well.
The approach also reflects growing industry recognition that standardized signals can improve transparency and reduce duplication across AI training pipelines.
Source: OpenAI Blog · Summarized by HeadlinesBriefing