Comparative performance and temporal variability of large language models on orthodontic questions from a national dental specialty examination

Abstract

This study compared the performance of three artificial intelligence–based chatbots (ChatGPT-4o, ChatGPT-4.5, and Gemini 2.5 Pro) on orthodontic questions from the Turkish Dental Specialty Examination at two testing time points. A total of 179 orthodontic multiple-choice questions from 18 examinations conducted between 2012 and 2024 were administered in Turkish to ChatGPT-4o, ChatGPT-4.5, and Gemini 2.5 Pro at two predefined time points (April 2025 [T1] and July 2025 [T2]). All models were tested under standardized conditions in newly initiated chat sessions, and questions were presented in the same format and order. Items were categorized by structure (text-based versus visual) and subject area. Statistical analyses included McNemar and Cochran’s Q tests; Bonferroni correction was applied for post hoc pairwise comparisons. At T1, ChatGPT-4.5 achieved the highest accuracy (82.12%), whereas Gemini 2.5 Pro performed best at T2 (89.94%). Gemini 2.5 Pro showed a significant increase in observed accuracy between T1 and T2 ( p < 0.001), while ChatGPT-4.5 demonstrated a modest increase ( p = 0.035). All models performed significantly worse on visually oriented questions than on text-based questions ( p < 0.05). Accuracy also varied across topic categories. Observed LLM performance differed across two testing time points under the study conditions, highlighting the need for caution when interpreting repeated assessments conducted through evolving commercial interfaces. The limited number of visual questions precludes definitive conclusions about multimodal performance; however, the lower observed accuracy on visually oriented items warrants further evaluation using larger and more diverse visual datasets.