Toward on-premises RAG pipelines for AI-assisted neuroradiology: practical considerations on privacy, accuracy, and timeliness

Abstract

Accurate differential diagnosis (DDx) in neuroradiology is challenging and remains vulnerable to cognitive biases and inter-reader variability in clinical practice. Despite expert training, human-generated DDx may be limited in breadth. Large language models (LLMs) have shown promise in assisting DDx generation; however, their performance may be limited by incomplete domain knowledge and concerns regarding data privacy and deployment latency. Retrieval-augmented generation (RAG) offers a potential solution by integrating external knowledge bases to enhance diagnostic accuracy while enabling on-premises implementation. To determine whether RAG improves LLM performance for neuroradiologic DDx and to compare proprietary versus open-source systems. We assembled a dataset of 737 challenging neuroradiology cases. Each case provided free-text imaging findings, clinical information, and a ground-truth diagnosis. A 1195-entry brain DDx knowledge base was synthesized using an LLM. Seven LLMs (five open-source, two proprietary) each generated ten-item DDx lists with and without RAG. Across models, RAG significantly improved DDx accuracy for the majority of LLMs ( p < 0.05). Among proprietary systems, O3 + RAG achieved 72.9% (Top-1), 85.8% (Top-3), 88.7% (Top-5), and 92.3% (Top-10). Among open-source options, Gemma-3 + RAG provided a strong accuracy–efficiency–privacy balance at 49.4% (Top-1), 71.4% (Top-3), 79.2% (Top-5), and 87.1% (Top-10). Coupling LLMs to a curated neuroradiology knowledge base via RAG markedly increases diagnostic accuracy and reduces the performance gap between open-source and proprietary models, while maintaining rapid, privacy-preserving inference suitable for clinical integration.