Independent Verification as a Measurable Property of AI Coding Pipelines: Three Pre-Registered Studies

Abstract

Evaluations of AI coding agents measure capability: can the agent solve the task? This paper studies an orthogonal property that decides whether an agent's output can enter a codebase without human inspection — whether the evidence accompanying a change is independent of the change itself. Three studies are reported, each with its scoring rules committed to a public repository before its data existed. (A) Controlled: on a frozen, seeded, held-out set of 50 SWE-bench Lite tasks, one proposer model silently shipped 17/50 wrong fixes (34.0%, 95% CI [21.2, 48.8]) when ungated, against 0/50 (CI [0.0, 7.1]) behind an independent verdict layer (Fisher exact p ≈ 3.0×10⁻⁶), at a delivered-correct yield of 24 versus 33. On 15 of those 17 tasks the gated arm produced a wrong patch as well: the layer did not make the model correct, it changed what happened to the error. (B) Observational: across 2,933 merged pull requests from 147 top-starred public repositories, 24.1% (CI [22.6, 25.7]) modify source and replace an existing test expectation in the same change; blind review puts the detector's precision at 6.9% (CI [1.9, 16.7]), implying an underlying rate near 1.7%. (C) Self-reported: of 1,190 closing statements from eight vendors' agents on SWE-bench Verified, labelled blind against the benchmark's hidden tests, a statement claims success on 86.9% of failed tasks and 87.7% of solved ones; sixteen intervals on the difference all cover zero. Exactly one statement of 1,190 reports failure. Artifacts for all three studies, including the pre-registered contracts and scripts that recompute every figure, are public at github.com/kolesnikov-arch/patchward.