Mercor says AI benchmarks cover 10 percent of US work hours

Anyone picking an AI model by its benchmark score is almost certainly reading a report about work the model is not being asked to do. Mercor, the hiring and research company, has mapped 29 widely cited AI benchmarks against the US Department of Labor's O*NET occupation taxonomy and says only 10 percent of American working hours sit on tasks the benchmarks cover.

The research page, published this week and posted to Hacker News with 339 points at the time of writing, counts the overlap task by task. ONET lists 923 occupations in the US workforce and 18,797 tasks across them. Mercor's initial index of 29 benchmarks maps to at least 5 prompts in 208 of those 923 occupations. Six of the 22 ONET job families have no occupation covered at that depth.

Where coverage sits and where it does not

Mercor's Benchmark Map shades each occupation by how many of its O*NET tasks appear in a benchmark. Computer and Math scores highest, with 29 of 36 occupations covered. Business and Financial comes next at 33 of 48, then Community Service at 10 of 14 and Legal at 5 of 7. At the other end, Science covers 15 of 60, Healthcare Practitioners 12 of 89 and Education and Library 9 of 62.

Mercor also plots coverage against each occupation's annual wage bill and reports that benchmarks lean toward the largest labor markets. In their example, benchmarks cover AI-exposed lawyers' tasks heavily but barely cover the tasks O*NET assigns to legal secretaries or court clerks, who are also AI-exposed by the Eloundou et al. (2023) tags Mercor used. The company writes that the long tail of smaller, middle-income knowledge work jobs appears undercovered.

How the map was built

Mercor says it picked 29 benchmarks for quality and economic relevance, with at least 25 publicly available tasks each, plus Mercor's own APEX benchmarks where the company has access to held-out prompts. The included benchmarks ask models to perform agentic tasks rather than answer questions. Coverage probabilities come from a task-level logistic regression on AI exposure and log wage bill, with error bars that account for clustering at the occupation group level.

What a buyer can do with this

Mercor's data is self-reported and the sample of 29 benchmarks is small, so the 10 percent figure is a map of coverage, not a verdict on model ability. The useful move for a team is to open the page, find the occupation closest to the work the model will do, and read how many of its tasks have any benchmark behind them. If the answer is zero, a vendor score for that model is a score on a different job, and the only honest test is the one a team runs on its own tasks.

Source

A benchmark map of the US economy, Mercor Research. Hacker News thread.