Who checks the answer key? Ground-truth defects in eight professional-practice benchmarks

Abstract

Benchmarks are known to contain incorrect ground truth, and models are already used to find it. That work concerns sparse annotation errors on individual instances, found statistically or by re-annotation. We report a class it does not reach: systematic errors in the program that derives a reference solution, which displace every case on the same code path identically. No correct majority remains for statistical detection, and re-annotation reproduces the error whenever it follows the same specified procedure. Such an error is equally invisible to the benchmark's own machinery: the grader compares against the reference, calibration recomputes it along that procedure, and adversarial cases are generated from it. All three are functions of the reference, and none can detect an error in it. Evidence comes from eight benchmarks — 33 tasks spanning CAD, quantity surveying, geotechnical assessment, tolerancing, BIM, customs classification and tax law — built to one design in which inputs, procedure and reference solutions all come from national standards and administrative rulings. Fifteen defects were found on the examination side rather than in the answers, and fourteen by the solvers, which hold the raw inputs but not the answer key. In the sharpest case a reference integrated a geotechnical index below the bottom of the borehole, placing 60 percent of a reported value in undrilled depth; it passed the grader, passed calibration to a difference of zero, and passed twelve of twelve adversarial cases. Version 1.1.0 adds a defect that was reachable by neither detector. A graded question scored a field the task statement never mentioned, so no solver could report its omission, and the reference was not itself wrong, so no check flagged it. Our adversarial suite had already constructed that exact submission and classified it as an attack, scoring it 94.6/100 — the same score, on the same question, that all six honest solvers received. This sharpens the paper's central claim: the three checks are functions of the reference, but the task statement is not, so none of them can detect a disagreement between the two. A benchmark can be internally consistent to the last decimal place and still be grading answers to a question it did not ask. The same design measures how far three documented effects vary by domain. Supplying the governing material ranges from moving a task 25 to 97 out of 100 to changing none of 61. Where scores tie, cost does not: in two domains replicated at n = 3 per configuration, scores tied while token consumption separated by 2.11x and 1.49x, at 15.6 and 13.9 pooled standard deviations. In one of the two, the configuration that is cheaper in tokens is more expensive in wall-clock time, so a single cost column cannot rank them. Answers change under identical repetition in 0 to 82 percent of cases, and half the unstable cases score identically across five runs by five different routes. Preprint. Not peer reviewed. The eight benchmarks it describes are archived separately and are listed in the related identifiers. Version 1.2.0 reports a fourth and fifth kind of examination-side defect, both of which we introduced ourselves. Repairing the task-statement defect of v1.1.0 produced a note that printed the answers to four of the repaired task's thirteen questions; all three solver runs reported it, and five checks — calibration, external cross-check, adversarial suite, a new scope check written for the previous defect, and a hash freeze — passed the statement carrying them. Auditing all eight benchmarks for the same fault then found it in two more, flagged by no solver: a supplied format description reproducing four rows of its own reference, and worked examples that were themselves graded answers. During one frozen run the operator became the defect, misreading a slow transport as a broken one and instructing two running solvers to truncate; the one that complied lost six of thirteen questions, and the one that refused reported the misdiagnosis. A freeze constrains artefacts and says nothing about the operator's conduct during a run. Two reported figures are narrowed. The largest material effect in the series (25 to 97 out of 100) is now doubly qualified: the unsupplied arm's failure does not replicate — 26, 26 and 92 over three runs, the spread turning on one binary guess — and the supplied material contains four rows of the reference, so 'explaining the format' and 'handing over answers' are confounded. A claim that one cost comparison was void is withdrawn: re-running that arm with the faulty condition text corrected cost three per cent more, inside its own run-to-run spread. The correction we were about to publish was larger than the effect it corrected. Cost at tied scores is now measured in three domains rather than two, and the instructions given to solvers are recorded as absent from six of the eight repositories. Version 1.2.1 resolves one of the two qualifications v1.2.0 placed on the series' largest material effect. v1.2.0 reported that the material supplied to the format-informed arm contained four rows of the reference solution, so that 'explaining the format' and 'handing over answers' were confounded. The confound has been measured rather than argued: the leaking document was left in place, a second copy was issued with every example value and entity number replaced and the explanatory content unchanged, and the arm scored 100 in all three runs, exactly as before. The disclosed rows were not carrying the score. Both limits on that inference were declared before the runs — the replacement changed two things rather than one, so only the absence of a drop is interpretable, and 100 is a ceiling — and both are stated in the text. The other qualification, that the unsupplied arm's failure does not replicate, is unchanged. Version 1.3.0 replaces an interpretation with a controlled experiment, and corrects two of the paper's own figures. The examination-side defect reported in v1.1.0 was diagnosed as the solvers answering what was asked; that was a causal claim and it had not been measured. Three successor tasks that share the reference byte for byte vary the two disclosure routes independently — whether the worked example shows the graded field, and whether the question text asks for it — completing a 2x2 with the original. Either route alone restores full agreement (100.0 in every cell but the original, which scored 94.6), so 5.4 points of a published score were a property of the examination rather than of the solvers. A solver identified the design's limit: the task naming the omitted key makes the condition 'asked, with the name available' rather than 'asked alone', and this is not removable, so the factor actually varied is whether the graded field's existence and name are discoverable at all. This does not soften the paper's central claim — the 2x2 was only constructible because replication had already exposed which field was affected. Controlled substitution can localise an examination-side defect; it cannot find one. Two corrections. The series is 33 released tasks, not 24, and the paper previously gave that total without stating what it counted; a new section fixes the rule — 25 distinct questions plus 8 variants that hold the question fixed and vary one factor — and readers estimating coverage should use 25. Separately, the same deduction appeared in the paper as both '4.8 points' and '5.4 points'; these are different scales (4.8 raw points out of the graded question's 15, which is 5.4 on the reported 100-point scale), and the text now uses the reported scale throughout. A new threats subsection records a defect class absent from the paper's own catalogue: a check that mutates the thing it checks. A hash freeze added to protect five published reference drawings rewrote all five, because the drawings are generated artefacts whose bytes are not stable across regeneration. No reported figure moved, but bytes released under a DOI were changed by the act of checking that they should not change. Every class in the paper's catalogue concerns a check that failed to see something; this one saw correctly and changed the subject while looking.