Scaffolded Answers and Process Self-Reports: Two Small Evaluations of an Operational LLM Deployment

Abstract

External context can specify how a language model should respond, but a response that satisfies operational criteria need not accurately describe the process that produced it. We examine two small evaluations of one operational deployment. In EXP-001, scaffolded outputs scored lower on the accuracy of process self-reports than clean outputs under two scoring procedures (74.6 vs 90.4; 82.7 vs 96.0). In EXP-002, guided answers satisfied more predefined rule criteria than bare answers (mean 0.125 vs 0.875 on eight tasks). The studies use different tasks and endpoints and are not directly comparable; we present them as a bounded case study motivating separate evaluation of answer quality and process-report accuracy. Headline scores were recomputed from the stored scoring files; a full reproducibility package is in preparation. This preprint narrows EFS v3.4 to its two completed evaluations.