Measuring Reverse-Specification Quality Without Ground Truth: A Three-Layer Framework and a Study of Scale-Induced Failure Modes in Autonomous Coding Agents

Abstract

Tool-using LLM coding agents can reverse-generate specifications from source, but the quality of those specifications is hard to assess when no reference specification exists. We present a ground-truth-free, deterministic-first framework that measures three layers - inventory coverage against a deterministically derived and independently validated unit population, reference soundness, and confidence calibration - deciding mechanically whatever the code determines and reserving an independent judge for the semantic residue. We apply it, under a pre-registered protocol, to a well-prompted autonomous agent over four repositories spanning a 22x size range (2.8-62.3 kLOC, N=3 runs each). We find that the scale failure mode is omission, not fabrication. Inventory coverage collapses from 100% at 2.8 kLOC to 47% - and to 11% under a strict lower bound - at 62.3 kLOC, yet what the agent writes stays largely faithful: an independent judge finds only 0-4.4% of sampled claims unsupported, with no upward trend across scale, and confirms 91% of the agent's top-confidence claims. Stated confidence is blind to the collapse: the share of top-confidence markers stays flat near 90% while coverage halves. A prompted agent's hazard at scale is thus not a specification full of falsehoods but one that confidently describes a shrinking fraction of the system. We report a pre-registered hypothesis of scale-induced false confidence as refuted, and locate the baseline's weakness precisely enough to motivate a controlled comparison against a structured methodology. Version note (2026-09-22). This version removes "Pre-Registered" from the title and states explicitly what the pre-registration record does and does not establish: the protocol was frozen before the confirmatory runs and its commit trail is published, but it was never lodged with an external registry, so the ordering cannot be independently certified. This version also names the deterministic inventory extractor with its pinned commit, cites the author's earlier methodology preprint, and points to a full replication package deposited at doi:10.5281/zenodo.22896753. Revision note (2026-09-22, second update). The wording of the manuscript has been made consistent with the caveat stated above: the terms "pre-registered" and "pre-registration" have been replaced throughout by "pre-specified", "fixed before the confirmatory runs" and "declared in advance". The protocol was indeed specified and frozen before the confirmatory runs, and the commit trail is published in the replication package, but it was never lodged with an external registry; the manuscript therefore no longer uses a term that implies third-party certification.