Longitudinal benchmarking of artificial intelligence models for the differential diagnosis of oral mucosal lesions: a controlled clinical validation study

Abstract

Abstract To perform a controlled longitudinal benchmarking analysis of contemporary artificial intelligence systems for the differential diagnosis of oral mucosal lesions and compare their performance with historical benchmarks obtained using the same biopsy-confirmed dataset and an oral medicine specialist. Using an identical biopsy-confirmed dataset of 100 oral mucosal lesions, multiple contemporary AI systems were assessed using a standardized prompt protocol. Diagnostic accuracy was defined as inclusion of the histopathologic diagnosis within the top three differential diagnoses. Performance metrics included overall accuracy, sensitivity and specificity for malignant lesion detection, inter-model agreement, and category-specific diagnostic performance. Results were compared with previously published specialist and ChatGPT-4 benchmarks. The oral medicine specialist achieved the highest overall diagnostic accuracy (70%), followed by the evidence-grounded platform OpenEvidence (66%), which approached specialist-level performance. General-purpose language models demonstrated heterogeneous performance, with accuracies ranging from 7% to 51%. Several models demonstrated high sensitivity for malignant lesion detection (up to 100%), although this was frequently accompanied by reduced specificity. Inter-model agreement analysis revealed clustering within model families and substantial divergence among lower-performing systems. Category-level analysis demonstrated stronger performance for malignant lesions and reduced accuracy for reactive and oral potentially malignant disorder categories. Despite rapid advances in AI development, diagnostic performance improvements remain uneven and model dependent. Evidence-grounded systems show promising progress toward clinical utility, while general-purpose models demonstrate variable reliability. AI systems may support oral lesion triage and differential diagnosis generation but should currently be integrated as adjunctive tools under specialist supervision, particularly in high-risk oncologic contexts.