The authors mapped 29 selected AI benchmarks to the O*NET taxonomy, which enumerates 923 U.S. occupations and 18,797 constituent tasks, to quantify how much of American work is represented in current evaluation suites. Using an LLM judge to tag each benchmark prompt with corresponding O*NET tasks—and validating those tags with human experts on Mercor’s APEX subset—they deemed a task “covered” if it received at least five prompt mappings. Combining this coverage with Hatgis‑Kessell et al.’s LLM‑derived estimates of each task’s share of working time revealed that only 208 occupations (≈23%) have any task meeting the five‑prompt threshold, six of the 22 O*NET job families have none, and roughly 90% of U.S. working hours are spent on tasks with little or no benchmark representation. The analysis further shows benchmark coverage is skewed toward AI‑exposed tasks (≈6× more likely) and toward tasks rated highly important by domain experts (≈6–10× more likely), reflecting a focus on large‑wage‑bill occupations such as software engineering and financial analysis. Initial occupation‑level scoring with APEX benchmarks for investment banking, law, accounting, and software engineering found that O*NET‑weighted averages differed little from simple prompt averages, but noted that less than one‑third of relevant tasks had sufficient APEX coverage, highlighting the need for broader, higher‑quality benchmark sets to reduce bias and capture the long tail of middle‑income knowledge work.

Read original