Web Fovea, a paper by Jiangang Han, shows that a mismatch between screenshot and browser coordinates causes clicks to land 3/4 of the intended distance. The service resizes screenshots before the model sees them, so a 1920 × 1080 image becomes roughly 1440 × 810, shrinking each pixel to 75 % of its true position.
I built a tiny site with four tasks and used Claude Haiku 5.5 to generate 80 successful decisions. After deliberately breaking the browser code, I replayed each decision and measured whether automated checks detected the fault and whether the combined model‑plus‑code agent could still finish the task. While spotting the error was straightforward, overcoming it required additional fixes; the agent’s score rose from 31.0 to 57.0 out of 100 as the four submissions were merged. The test involved 40 variants of each page, ensuring varied live‑website conditions.
The findings highlight a critical gap for developers relying on AI to navigate websites: detecting a mis‑click does not guarantee correct execution. This gap matters to web developers and anyone who builds sites via AI description. The paper’s code, logs, and figures are publicly available, underscoring the need for precise coordinate handling in AI‑driven web interactions.
Source: Towards Data Science · Summarized by HeadlinesBriefing