HeadlinesBriefing favicon HeadlinesBriefing.com

AI Crawler Blocking Low, AI Citations Rare

Hacker News •
×

8.9% of sites block at least one AI crawler in robots.txt, yet 94.8% of audited sites never surface in an AI answer. The AI Visibility Index tracks answer presence and machine readability separately, because being visible to AI involves two distinct problems.

Across 193 sites and 5,978 assistant answers, only 10 sites were named even once. Gemini led with 2.9% of answers naming the audited business, ahead of Chat GPT (1.7%), Claude (1.6%) and Perplexity (1.6%). 39 of 436 sites disallow an AI user‑agent; most block training crawlers—GPT Bot by 38 sites and Claude Bot by 36 sites—while retrieval agents such as OAI‑Search Bot and Perplexity‑User are blocked by only 4 and 3 sites.

54.6% of sites publish any schema.org structured data, but only 19.3% use Local Business schema. The presence data cover July 23–August 2, 2026; readability data cover May 16–August 2, 2026, gathered via Website Auditor’s free scanner. Blocking retrieval agents makes a site uncitable, whereas blocking training crawlers carries no cost in today's answers.