Do Professional Evaluation Criteria Improve LLM Analytical Artifacts? A Controlled Study with Source-Grounded Downstream Reconstruction

Abstract

This controlled study tests whether an explicit author-elicited professional evaluation frame improves one-shot LLM analytical memos beyond a length- and detail-matched generic expert control when the model, source evidence, task contract, output format, and execution policy are held fixed. Across 12 unseen public-company analytical cases generated with deepseek-v4-flash, the publication-facing primary endpoint was Source-Grounded Downstream Reconstruction (SGDR), a study-specific source-grounded downstream reconstruction instrument scored at the case level. A complete audit of the 114 source-grounded answer keys identified three source-identity defects; correcting those keys using the already-frozen reader responses yielded 4 professional-frame wins, 3 generic-control wins, and 5 ties, with a mean paired SGDR difference of +0.0111. The two-sided exact sign test was p=1.0000 and the exact sign-flip sensitivity test was p=0.703125, providing no clear evidence of superiority. A separate integrity audit found authority failures in both conditions, while qualitative mechanism review suggested targeted behavioral shifts that did not reliably translate into higher SGDR. The study uses a post-generation, pre-outcome-comparison evaluation amendment with prior mapping access by the orchestrator, and exact masked-reader runtime provenance is incomplete. The contribution is therefore a bounded treatment-and-evaluation design and a negative/inconclusive boundary result, not a claim that professional judgment was transferred to the model or that explicit criteria are generally ineffective.