A fisheries estimator from the 1890s, tested on 294 CUAD contracts, landed within 2 points of true recall for most agents with a 10% hand audit. The article explains how capture-recapture methodology can estimate the completeness of AI agent evaluations. The author applied this statistical method to map conifer forests, comparing Open Street Map and Ordnance Survey data to reveal gaps in coverage.
By treating each map as a "catch," the overlap between sources determines the estimated total. The method assumes independence between sources; violations, such as famous forests appearing on both maps, can bias results. A small check on 18 well-known forests showed 56% overlap, suggesting specialization between volunteer and national survey data.
The technique offers a practical way to quantify unseen gaps in AI benchmarking without exhaustive auditing.
Source: Towards Data Science · Summarized by HeadlinesBriefing