Over 28 blind debugging experiments on real bugs from GStack, immer, and decimal.js reveal that AI coding agents struggle not with complexity but with missing information. Two difficult bugs — deep in Immer's internals and a numerical edge case in decimal.js — were fixed correctly in all 16 attempts. However, a seemingly trivial HTTP client bug in GStack that silently dropped a retry option defeated every model and workflow across 12 runs.
Each AI-generated fix corrupted user data yet passed the full 84-test retry suite. The failure wasn't detection — a reviewer agent correctly identified the data corruption risk but approved the change anyway, judging the issue unlikely. This highlights a critical blind spot: when fixes depend on undocumented API contracts, AI lacks the judgment to reject plausible but wrong solutions.
Difficulty didn't predict failure; missing information did. For teams merging AI fixes based on green CI, this is the exact failure mode that slips through — simple-looking bugs with passing tests that quietly corrupt data in production.
Source: Towards Data Science · Summarized by HeadlinesBriefing