Quality optimization with low positive factual hallucination for the HPI in hyperthyroidism admission notes using multi agent LLM with RAG

Abstract

Admission note quality affects clinical safety and research reliability, yet resident drafts often suffer from missing information, unclear logic, and other issues, requiring time-consuming manual review. Current large language models (LLMs) focus on text generation or formatting correction but rarely optimize clinical logic or mitigate hallucinations in specialized notes. We developed HQ-ANR (High-Quality Admission Note Refinement), a hyperthyroidism-specific quality-control model that integrates multi-agent collaboration with diagnosis-driven retrieval-augmented generation (RAG) to optimize the history of present illness (HPI) in admission notes. Built on a general-purpose LLM (Qwen3-32B), HQ-ANR anchors generation to a dual-source knowledge base (guidelines and expert notes) and employs specialized agents for symptoms, examinations, and treatments. In a blinded evaluation of 30 resident-drafted hyperthyroidism notes, HQ-ANR significantly improved median Physician Documentation Quality Instrument (PDQI-9) scores (from 39.00 to 42.00) and reduced the hallucination rate (positive factual hallucinations only) to 6.67%, compared to 26.67% for a general LLM with chain-of-thought (CoT) prompting. Efficiency analysis showed HQ-ANR reduced physician refinement time by an average of 112.9 s per note ( p < 0.001). These results demonstrate that augmenting a general LLM with external knowledge and a multi-agent framework can achieve reliable, in-depth optimization of clinical notes without expensive vertical training, offering a practical pathway toward intelligent, safe documentation quality control and reducing physician documentation burden.