Modeling Reliability and Validity Across OSCE Station Numbers in Final MBBS Exam

Abstract

Background: The Objective Structured Clinical Examination (OSCE) has been the backbone of assessment in medical education, and the number of stations required to achieve sufficient reliability in high-stakes examinations is an important consideration for resource-limited programs. Methods: The correlation between the number of stations and psychometric performance was simulated using post-hoc resampling of the 17-station Final MBBS Medicine and Therapeutics OSCE across three campuses of the University of the West Indies. Blueprint coverage was maintained in balanced subsets of 6, 8, 10, 12 stations. We compared the failure rates of subset and full-exam failure, internal consistency, generalizability, and correlation of scores. Results: Failure rates increased as the number of stations decreased; however, none of the differences between subset and full-exam performance were significant. Station count was positively related to internal consistency (α ≈.32-61 at 6 stations; α ≈.64-70 at 17 stations) and G-coefficients (≈.18–.42 at 6 and ≈.36–.70 at 17 stations) but were still lower than the highly consistent ≥.80 criterion. High Construct validity even at 6 stations (with high subset total correlations (r≥.79), and approaching 1.00 with 17-station subsets, ANOVA showed a subset size effect on mean values for campuses 1 and 2 but not for campus 3. Conclusions: Ten to twelve stations maintained a strong construct validity. The reliability at 10-12 stations was close to that of the 17 stations total. Campus-level variability highlights the need for examiner training, station quality, and better standard setting in addition to the station count.