A practical guide to automated test assembly for educational evaluation in health professions: a demonstration using the R package ROI

Abstract

Automated test assembly (ATA) using mixed-integer programming (MIP) is standard practice in international testing organizations, but detailed and accessible practical guidelines remain scarce for health professions assessment. This study introduces the mathematical foundations of MIP-based ATA and provides a step-by-step tutorial demonstrating its implementation with the open-source R package ROI and the HiGHS solver. We constructed a modified 928-item pool with empirical item parameters from the Clinical Medical Science Comprehensive Examination, establishing 31 content categories and 18 enemy-item groups. Using the ROI package and the HiGHS solver, we modeled and solved 3 assembly scenarios: a single 80-item form under a relative target, 2 parallel 80-item forms under a relative target, and 2 parallel 80-item forms under an absolute target combined with additional mean item difficulty constraints. All 3 optimization scenarios satisfied the test-length, content-blueprint, and enemy-item constraints. The relative-target forms attained observed-to-target information ratios within rounding error of the targets, producing closely equivalent parallel forms. The absolute-target forms closely approximated the specified information curve, with a maximum deviation of 0.27 information units, and their mean difficulties fell within the targeted range of 0.2 to 0.4. All formulated MIP models converged to optimal solutions within a few minutes. The practical combination of the ROI package and the HiGHS solver offers an efficient, reproducible, and cost-effective method for test assembly. This approach may enable Korean health professions examination programs to construct psychometrically equivalent and content-balanced test forms, facilitating the transition toward item-bank-based testing systems.