Checking AI-generated clinical documentation against its source record: eleven sealed evaluations of a deterministic checker and an exploratory language-model second reader

Abstract

Background. AI-generated clinical documentation may preserve much of its source wording while altering factual content, temporal status or evidential qualification. Deterministic checking offers reproducible comparison against the source, but its performance and limitations on free-text clinical documentation are uncertain. This study evaluated a rule-based checker across sequential sealed synthetic corpora and explored whether a local language model could add detection when used as a second-stage reader. Methods. A synthetic pre-operative assessment record, developed through three versions, provided the source. A commercial AI documentation system generated notes from the record text alone. Seven sequential sealed evaluation corpora, each of 100 notes written by language-model writers given the record and, except for one writer in the seventh corpus, fifteen distortion definitions, were scored once against checker versions frozen before each corpus was written. The checker applies six primary labels and eleven supplementary deterministic checks. Outcomes were faithful lines flagged (false warnings), planted distortions missed, misses within a pre-specified consequential category, and binary classification accuracy. The seventh corpus included writers of three model classes and one writer operating under a plain clinician's brief without distortion categories. An exploratory experiment, specified before any verdict was read but conducted on that existing corpus, compared the checker alone, a local 35-billion-parameter model alone, and two serial configurations. Results. On the seventh corpus the six-label configuration achieved 82.2 per cent binary accuracy, detecting 44.9 per cent of planted distortions and 67.0 per cent of pre-specified consequential distortions, with 3.0 per cent false warnings. Adding the supplementary checks raised these to 86.2, 80.4 and 91.7 per cent, with 11.5 per cent false warnings; disagreement between the checker and one writer's faithful labels, identified by author review, accounts for most of that figure, and the rate was 6.0 per cent in a sensitivity analysis excluding that writer. Across the seven corpora the planted-distortion miss rate fell from 32.0 per cent in the first evaluation to 19.6 per cent in the seventh, reaching a low of 19.4 per cent in the sixth, and the pre-specified consequential miss rate fell from 23.3 per cent in the second evaluation to 8.3 per cent in the seventh; the final cycle added eight detections among 494 planted distortions (1.6 points) and no change in consequential detection. Errors from the plain-brief writer were missed at 30.1 per cent; the largest single class of those misses, a quarter, was a fact, action or finding invented inside an otherwise faithful sentence. A language model restricted to lines the expanded checker had passed raised accuracy to 87.2 per cent and consequential detection to 99.1 per cent, with 14.0 per cent false warnings; the same model as a second opinion on the checker's flags, or alone, performed markedly worse than the checker. Conclusions. Deterministic source checking identified most planted distortions but remained limited by proposition-level changes that line alignment and word overlap do not reliably identify. A serial language-model reader added detection at the cost of more false warnings. Findings are restricted to synthetic material from one clinical record, with author-dependent reference assessment and writers from one model family. Independent clinical validation, and evaluation of effects on clinicians, remain outstanding. Version 1.4 (11 October 2026): adds an eleventh sealed corpus scoring a filter on the second reader's verdicts (omission-only verdicts set aside), with no rule change; version 1.3 adds a tenth sealed corpus written under a brief that settles the self-report marker disagreement on the rule's side, with no rule change; version 1.2 adds a ninth sealed corpus scoring four fixes to the checker's own false-warning classes found in the eighth; version 1.1 adds an eighth sealed corpus written after two post-development rule fixes, with the page, tables and second-reader experiment updated to it; version 1.0 reported seven corpora. The live checker, its figures and its evaluation manifests are public at https://fellowship.narrativelogic.co.uk/#record-silent. The vendor of the commercial AI documentation system is not named.