A new analysis finds that 90% of American working hours are spent on tasks with little or no coverage from existing AI benchmarks. The study maps the US labor force, which the US Department of Labor's O*NET taxonomy divides into 923 occupations grouped into 22 job families, against an initial list of AI benchmarks.
The results show that only 208 of 923 occupations have a task covered by at least five benchmark prompts, and six of the 22 job families have no such occupations. This makes it difficult to judge how well AI models perform the work Americans actually do.
Coverage is also uneven by economic weight. Benchmarks heavily cover tasks of AI-exposed lawyers, but barely touch those of similarly exposed legal secretaries or court clerks. The researchers say the long tail of smaller, middle-income knowledge work jobs appears undercovered, suggesting the AI community is focusing first on the largest labor markets.
The researchers used 29 benchmarks selected for public availability, agentic tasks, economic realism and economic diversity. The analysis covers 18,797 O*NET tasks across all occupations. The team says it hopes to expand coverage of model capabilities across the economy and fill more gaps in understanding what AI models can do.
Source: Hacker News · Summarized by HeadlinesBriefing