When Retrieved Evidence Is Not Enough: Measuring Evidence-Following in Retrieval-Augmented Generation

Abstract

We study the generator half of RAG in isolation: evidence is placed in the prompt rather than retrieved. Retrieval-augmented generation is usually judged by whether the retriever found the right documents. That misses a second failure: the generator can ignore the evidence, fill gaps from prior knowledge, or resolve contradictions silently. We introduce a controlled synthetic benchmark of 105 agricultural questions (15 of each of seven epistemic conditions): sufficient evidence, insufficient evidence, conflicting prior, conflicting evidence, irrelevant context, partial evidence, and attribution. Evidence is placed in the prompt, so the experiment measures following, not retrieval. Three models were run on all 105 cases with a normal helpful prompt and an evidence-constrained prompt (630 answers). The author labelled every answer, blinded to prompt identity. A case passed only if the expected behaviour was met and the answer added no unsupported claim. Constrained prompting passed 103/105, 103/105, and 96/105 on Gemini 3.5 Flash-Lite, gpt-4.1-mini, and gpt-oss-20b; the normal prompt passed 57/105, 35/105, and 9/105. The gap is concentrated where the model should abstain or flag conflict. Agriculture is the testbed, not the scientific claim. The author is the sole rater.