Validation of a Pre-CS1 Concept Inventory

Abstract

As programming has become a prominent component of K–12 education in large parts of the world, incoming university students increasingly enter higher education with prior programming experience. To inform instructors of how far students have already progressed into the introductory programming (CS1) curriculum at the point of entry, we developed the Prior Informatics Knowledge Assessment (PIKA), a diagnostic assessment tool targeting elementary CS1 concepts. This paper presents a validity argument for PIKA as a concept inventory by integrating multiple sources of evidence. Test content validity is supported through curriculum triangulation and iterative expert review. Response process evidence derives from think-aloud interviews ( \(n=10\) ), examining whether items elicit intended conceptual reasoning. Internal structure is evaluated using confirmatory factor analysis and item response theory on large-scale response data ( \(n=1{,}355\) ), assessing dimensionality and item function. Evidence based on relations to other variables is provided by correlating PIKA scores with CS1 final examination performance ( \(n=280\) ). Results indicate that PIKA responses are largely governed by a single latent trait corresponding to introductory programming knowledge. Model fit indices support an essentially unidimensional structure, and think-aloud findings suggest that most tasks require relevant CS1 knowledge to be answered correctly. PIKA scores show a moderate positive correlation with CS1 exam performance, indicating that PIKA measures knowledge that meaningfully relates to CS1 learning outcomes. At the same time, the analyses reveal limitations. Certain constructs—particularly the use of for -loops with lists—appear underrepresented in some K–12 curricula, limiting the diagnostic value of multiple items targeting this content. In addition, item response theory results and response distributions suggest a shortage of difficult items, reducing PIKA's precision for students with advanced prior knowledge. These limitations will be targeted for refinement in upcoming PIKA administrations and changes will be studied in future validation work.