Accountability without an Oracle: ScholarHound for LLM-Produced Scientific Commitments, and Why Claim-Level Gold Is the Binding Constraint

Abstract

Scientific language-model systems increasingly retrieve papers and write evidence-backed syntheses, yet their judgments are checked by the same class of models that produced them. We present ScholarHound, an accountability layer separating model-mediated interpretation from authorization to update persistent research state, and report two evaluations. On SciFact claim–abstract labels, a frozen quorum aggregator did not improve on five-model voting: 2/19 committed errors direct, 2/18 gated. We then constructed a final-text digest-propagation benchmark whose hidden scoring receipts are public reviewer–author histories from eLife and Nature Communications, and evaluated the original claim-boundary gate on 16 cases (4 strict over-propagation risks, 12 safe controls). Across four model execution surfaces, direct 3-of-4 voting propagated all four risk claims as stated; gated voting propagated one, with three paired improvements and no reversals. The result is unchanged when the one surface with unverified backend identity is removed. Four paired items cannot establish a population effect, and gating raised no-consensus outcomes from one to three of sixteen. The second measurement carries the larger sample. Constructing that benchmark required screening roughly 27,000 records across four public sources through four packet revisions to obtain 16 executable items, with claim localization — linking a reviewer objection to a specific sentence and to its resolution in the final text — as the binding constraint. The census identifies benchmark construction, not model-call capacity, as the present ceiling. Against contemporaneous claim-extraction infrastructure now indexing millions of quote-bearing claims while evaluating groundedness alone, the yield gap spans roughly five orders of magnitude. All artifacts, including three discarded packet designs, are deposited. Working preprint, version 0.4. Not peer reviewed.