ObviousBench: The Cost of Not Making Obvious Mistakes

Abstract

Language models deployed in public-facing products still make mistakes any careful person would catch instantly: miscounting the letters in a word, botching a trivial spelling transform, ignoring a plainly stated constraint. These visible failures erode user trust faster than failures on hard problems. ObviousBench measures this failure surface directly: 144 private held-out items across eight human-trivial task families, scored by deterministic Python scorers (no LLM judge), run as bare chat-API calls (no system prompt). Because users expect obvious questions to work every time, the primary metric is answer pass^3: an item counts only if all three attempts are correct. We evaluate 343 configurations of 172 models from 14 providers (148,176 attempts, $151.17 API cost). Unlike capability benchmarks, ObviousBench is saturatable by design—an unsolvable item is a defect, not difficulty—and 16 configurations reach 100.0%, certifying the item set. The median configuration scores 90.3% and the weakest 17.4%. Configured test-time compute is the dominant lever: across 46 models with three or more reasoning-effort settings, the median within-model range is 17.0pp (maximum 59.7pp), while newer generations often regress or leave the cost-reliability frontier unmoved. Repeated attempts matter: 89.1% of item cells are right at least once, but only 80.8% survive all three attempts—and a 20-attempt extension on one small model shows reliability still falling past k=3, with higher effort flattening the decay. We release the harness, public example items, and full aggregate results, and position ObviousBench as a preflight and regression instrument that prices visible-mistake risk, not a global ranking. Project page: https://obviousbench.com — Code and public items: https://github.com/adamallcock/obviousbench