A study of retrieval-based AI systems found that the right amount of information and prompting method vary between question answering and code generation.
Researchers examined three basic decisions in retrieval-augmented generation systems: whether to use retrieval, how many documents to provide, and how to incorporate those documents into an answer. Their experiments covered question answering and code generation across three language models and six datasets.
The results differed by task. Question answering showed a consistent range of five to 10 useful documents, while code generation had no single best amount. Prompting methods also helped code generation more than question answering, indicating that system design needs to account for both the task and the language model being used.
How the tasks differed
The study found that retrieval should be used selectively rather than automatically. The recall level—the share of relevant documents successfully retrieved—needed to vary by situation, and some failure modes affected up to 12.6% of samples even when all relevant documents were available.
For question answering, five to 10 retrieved documents generally produced the best results across the tested datasets and models. Performance reached a plateau beyond 10 documents. For code generation, the best number varied from one to more than 16 documents depending on the model and dataset, so the researchers found no universal setting.
The way retrieved information was presented also mattered. Prompting methods produced significant benefits for code generation but little improvement for question answering. Their effectiveness depended on the model, task and method used; some approaches could also reduce performance in particular settings.
Why system design matters
The findings suggest that adding retrieval to an AI system is not automatically helpful, and that simply increasing the amount of retrieved information is not a reliable solution. Developers need to choose retrieval settings and prompting methods according to whether the system is answering questions or generating code, as well as according to the language model involved.
This provides practical guidance for deciding when retrieval is likely to add value and how much information to supply. It also indicates that strategies that work well for one task may not transfer to another.
Evidence and caveats
The researchers ran controlled experiments across three language models and six datasets covering question answering and code generation. To study when retrieval helps, they began with all relevant documents and progressively replaced some with distracting documents—documents that were related to the query but did not contain the answer. They also varied the number of retrieved documents and compared several prompting strategies.
The recall experiments used favorable conditions, including clean relevant documents and controlled ratios of relevant and distracting documents. They did not include complications such as contradictory information, system delays or incomplete knowledge bases, so the reported recall thresholds are upper-bound conditions rather than a full picture of deployment in practice. The studies of document number and prompting used only OpenAI's text-embedding-3-small retriever, which may limit how broadly those results apply to other retrieval systems.