A software engineer's deep dive reveals fundamental flaws in AI visibility measurement tools that promise precise rankings for brands in ChatGPT, Claude, and other answer engines. These platforms generate tidy metrics like mention rate and share of voice, but the underlying systems are inherently unstable and personalized.

Frontend scraping captures only one synthetic session with fixed account history, geography, and subscription tier. Real buyers asking different prompts from varied locations receive different answers, making single-sample measurements misleading. Cloud-based mass scraping introduces additional bias through concentrated IP patterns and repeated logins that synthetic accounts don't reflect.

API-based approaches offer better repeatability but measure different surfaces than consumer apps. Provider APIs lack the personalization, location inference, and product-specific features that real users experience. Meanwhile, vendors manipulate prompt sets and scoring formulas to manufacture headline numbers - the same brand can show 20%, 16.8%, or 31.4% share of voice depending on methodology.

Geography particularly undermines these measurements for local businesses. A single global visibility rank becomes meaningless when 'best commercial roofing company near me' produces location-dependent results. Without transparency into prompt lists, run frequency, and scoring weights, these dashboards show constructed metrics rather than measurable truth.

Canonry