We read the HTTP response headers of 217,097,223 page responses from Common Crawl, 10% of its September 2026 crawl of 2.17 billion pages, and looked for any header written for an AI system. 7,959 sites sent one. Almost all of them point agents to another resource, such as an llms.txt file or an MCP server. A small number try to give the agent orders. Whether any of it reaches your model depends on which parts of the response your fetch tool passes along, so the defense starts there.
Of those sites, 7,271 sent pointers, and 6,989 of them sent x-llms-txt. A pointer is the server telling your agent where to go next. Check that your agents treat those destinations as new, untrusted fetches. Just 30 sites sent injection attempts, 24 of them in a header named X-AI. Most were pranks. The few that ask for real actions only work when the agent holds the matching power, such as a shell or database access. In total, 283 distinct header names were aimed at AI.
Some sites use headers to push back on agents. Two put strings named after Anthropic's test triggers in protest-named headers, and one warns hacking agents away on its error pages. Expect your agents to meet text written to stop them as well as to steer them. A detector that matches on names alone would have flagged 910 sites by mistake, so alert on what the agent does after a fetch.
In October 2026 tests, Claude Opus, Claude Sonnet, and Claude Haiku running in Claude Code did not act on the test headers by default. When the user's prompt told them to follow the site's directions, they obeyed in 7 of 9 runs, so the refusal depends on how much authority your prompts give a site.
Source: Hacker News · Summarized by HeadlinesBriefing