Automatic Analysis of Dialogic Argumentation: An Exploratory Study on Performance of Large Language Models in Context of Secondary Mathematics and Science Education

Abstract

This study explores dialogic argumentation in secondary-level mathematics lessons and the potential of artificial intelligence (AI) and large language models (LLMs) to analyze these interactions. Its primary aim is to explore how and to what extent AI can assist in automating the complex process of analysing classroom interaction. Although argumentation and dialogic interactions share similarities, they differ in important ways: argumentation typically centers on justifying a single viewpoint, whereas dialogic interactions involve multiple perspectives, and not all dialogic interactions require justification. The study uses transcription of a secondary school mathematics lessons, employing human-validated coding protocols to create prompts for LLMs. Findings indicate that LLMs can reliably identify justifying moves but struggle to detect dialogic moves, often requiring repeated prompt refinements. The main challenge lies in clarifying the essential features of dialogic interactions for AI, highlighting the limitations and potential of automated analysis for educational research and practice.