Automated assessment of the nature of scientific inquiry (NOSI) using machine learning and transformer-based language models

Abstract

Background Effective assessment of the nature of scientific inquiry is foundational to scientific literacy but is constrained by the resource-intensive, manual scoring of open-ended instruments such as the Views About Scientific Inquiry.Purpose We addressed this challenge by developing and validating a comparative machine learning framework to automate and standardise the evaluation of Views About Scientific Inquiry responses responses from 503 Turkish pre-service teachers.Design/Methodology We employed a feature-based Random Forest regression model and a deep contextual transformer-based BERTurk classification model using 5-fold cross-validation.Results The Random Forest model established a reliable and stable methodological baseline, demonstrating strong predictive alignment with expert scores and minimal variability across repeated validation runs. Conversely, the BERTurk model, despite showing high initial potential, experienced a severe, non-reproducible collapse in performance during systematic stability tests.Conclusions This outcome highlighted significant operational fragility under conditions constrained by limited data and a morphologically complex language. This comparative analysis reveals a critical methodological tension in educational artificial intelligence: model stability and interpretability must take precedence over peak performance for responsible application. The Random Forest framework offers a transparent, scalable solution for the automated, highly consistent assessment of the nature of scientific inquiry and serves as a diagnostic tool for identifying specific linguistic and conceptual difficulties in scientific reasoning.