ChatGPT, Can You Generate Multiple-Choice Questions for an Undergraduate Computing Course? An Evaluation of its Effectiveness Using Classical Test Theory

Abstract

Educators are increasingly adopting large language models such as ChatGPT to generate multiple-choice questions (MCQs), yet empirical evidence on the quality of these items in computing education assessments remains limited. This study compares 30 ChatGPT-generated and 30 manually crafted items from an undergraduate software design course, and evaluates item difficulty, discrimination, and distractor quality using Classical Test Theory. We triangulated two evaluations using within-subjects and between-subjects designs. Our quantitative findings show that all three metrics in both evaluations point in the same direction. ChatGPT-generated items are easier, less discriminating, and exhibit poorer distractor quality than manually crafted items. Distractor quality of ChatGPT-generated items is significantly worse in both designs. Difficulty metric is significant in the within-subjects design and discrimination metric is significant in the between-subjects design. We discuss the quantitative findings for each evaluation and across the two evaluations. A qualitative analysis further identifies six construction patterns in ChatGPT-generated MCQs that contribute to non-functioning distractors and help explain their psychometric characteristics. These patterns provide practical guidance for instructors to review and revise ChatGPT-generated items before deployment.