A study of the top 10,000 websites' `robots.txt` files reveals a significant oversight in how sites manage AI crawlers. While many owners rushed to block training bots like OpenAI's GPT Bot in late 2023, they largely failed to implement rules for a second type of bot: the answer-time fetcher. These bots, used by AI services to retrieve page content for live answers, are often left unaddressed.

Analysis shows that 71% of sites blocking Anthropic's Claude Bot did not specify rules for its answer-time counterpart, Claude-User and Claude-Search Bot. Similarly, 53% of sites blocking OpenAI's GPT Bot neglected to address its answer-time fetcher, OAI-Search Bot. This suggests a misinterpretation of AI vendors' bot strategies, with most rules only accounting for one of the two distinct bot functions.

This gap creates a blind spot for publishers. While companies like Cloudflare and x402 are developing payment rails for answer-time fetches, the current `robots.txt` implementations do not support this transactional model. The overwhelming majority of sites still permit answer-time bots, but this is by omission, not by deliberate choice to allow referrals while denying training data. Only a tiny fraction, under 4%, actively block training bots while explicitly allowing answer-time bots.