Preregistration of the Expert-Panel Analysis Plan: LLMs as Pre-Administration Item Reviewers in Programming Education

Abstract

This record preregisters the analysis plan for the expert-panel component of a study comparing large language model (LLM) item reviews with classical item analysis for a 60-item multiple-choice C# programming achievement test (associate degree, Türkiye). The plan was registered before any expert response was collected. Classical item analysis (N = 150 students) and blind LLM evaluations (Claude Opus 5.5, GPT-6.1 Sol, DeepSeek V4 Pro; two rounds each) were completed before registration and are not part of the preregistered analyses. The plan specifies content-validity indices (I-CVI, modified kappa, S-CVI), inter-rater consistency measures, comparisons of expert judgements with LLM judgements and empirical indices, expert versus LLM accuracy in predicting the most attractive distractor, and a triangulation-based item-selection protocol. Item texts are not shared for test-security reasons. Version 1.1 (2 October 2026) adds an amendment registered before any expert response was received: panel of four experts; exclusion of an AI-generated pilot response from the preregistered analyses; and a sensitivity analysis by programming background. The primary analyses are unchanged.