Specialty-aware benchmark design, split integrity, and baseline evaluation for Korean medical question answering

Abstract

Korean medical question answering requires evaluation resources that reflect local terminology, clinical specialty structure, and essential-care priorities. We characterized the AI Hub Specialized Medical Knowledge and Essential Medical Knowledge datasets, audited the independence of their distributed validation labels, and implemented a package-specific baseline evaluation. The resources contained 222,073,009 source-data tokens and 34,487 labeled question-answer pairs across 17 labeled domains. Domain concentration was pronounced (Herfindahl-Hirschman Index 0.178; effective domains 5.62), and 78.7% of items were multiple-choice. Question-type distributions differed between datasets (χ²(2) = 165.82, p < 0.001; Cramér’s V = 0.069). Among 3448 distributed validation records, 3333 (96.7%) shared qa_id values with the model-training files, with median normalized question similarity 0.885, indicating that these files were unsuitable as independent tests of the accompanying fine-tuned models. On an AI Hub-supplied 2494-row KorMedMCQA-derived evaluation file containing 2489 unique normalized questions, essential-care and specialized-care Qwen2.5-14B models achieved 64.35% and 64.03% accuracy, respectively; the paired difference was not significant (exact McNemar p = 0.440). The framework is technically implementable, but valid specialty-level comparison requires a separately constructed, decontaminated test set and prespecified clinical error review.