plan-failure-bench: A Machine-Checkable Benchmark of How Language Model Planners Fail

Abstract

Evaluations of language model planners mostly rest on foundations a reader cannot audit: success rates that cannot tell a model refusing every task from one walking into every trap, ground-truth labels that are authored rather than proved, and human or LLM judges somewhere in the scoring loop. The contribution of this paper is methodological: a set of construction rules for planner evaluations that remove all three, demonstrated in plan-failure-bench. Ground truth is decidable by construction. Each of 60 instructions over two symbolic household environments either admits a valid plan or plants exactly one trap from a six-way taxonomy (unreachable goal, missing capability, ambiguous referent, precondition trap, sequencing trap, constraint trap), and every label carries a mechanical proof obligation re-run in continuous integration. Models answer in a small JSON action language with explicit infeasible and clarify responses, so trap detection is machine-checkable and never reported without its paired false positive count. A deterministic checker, differentially tested against an independent PDDL toolchain, assigns exactly one verdict per response; every instruction also exists in a semantically obfuscated condition applied as a versioned invertible renaming, so the checker only ever scores the canonical world; and the results tables are generated directly from the committed run records, so no number in the paper can drift from its data. The headline artefact is the confusion matrix between planted trap and observed verdict. A four-model grid, sized by free-tier access and reported as counts and hypotheses, demonstrates what the instrument yields. Models fail in distinct, stable ways. A 70B model's planning judgement survives semantic obfuscation essentially intact; an apparent execution collapse under our first token scheme was caught and retired by the obfuscation's own versioning. Over-refusal tracks surface semantics. For the smaller reasoning-generation model, unreachability detection survives obfuscation only where the isolation is stated rather than left to be inferred from topology. The first frontier reasoning model clears house_01 in both conditions with identical ideal diagonals, then repeats every headline count on the second environment in both conditions, with the same single sequencing seed failing identically in each; for that model, semantic removal changes nothing this benchmark can measure, which bounds every failure finding here to smaller and non-reasoning models. This is a working paper. Code, environments, seeds, and every run record are public at https://github.com/munawarkazmi/plan-failure-bench.