Evaluation and analysis of teacher-student interaction efficacy based on multimodal machine learning

Abstract

The quality of teacher-student interaction is crucial to the achievement of learning outcomes, engagement and cognition of students. By using multimodal machine learning, we can gain a more comprehensive and accurate understanding of the interactions taking place by analyzing multiple behavioral and emotional cues. Current approaches, however, are not always able to handle data that is not structured and is difficult to adapt to real time, and they are not always able to provide personalized feedback mechanisms. To solve these issues, this work introduces a Multimodal Deep Learning Framework using Attention-Based Fusion (MDLF-ABF) which can successfully fuse visual, auditory and textual modalities. An attention-based fusion mechanism emphasizes salient features of the modalities, thereby allowing a dynamic and contextually appropriate understanding of classroom interaction. The proposed framework is implemented in a real-time engagement detection system that tracks the behaviors, emotions, and communication indicators of students and provides teachers with useful feedback. Experimental testing with real classroom datasets show that MDLF-ABF significantly outperforms the detection accuracy, contribution accuracy, personalization index, precision, recall, F1 score and Cue recognition by 89.5%, 80%, 0.83%, 88.2, 87.5, 87.8, and 85%, respectively, from baseline unimodal and early fusion models. The results indicate improvements in student engagement scores, greater ability to adapt to learning styles, and greater effectiveness of instruction. The proposed MDLF-ABF framework demonstrates high multimodal engagement detection accuracy with a Personalization Index of 0.83, an absolute metric value that represents an effective adaptation to the individual level’s behavior patterns, ensuring consistent reporting of detection results even across different multimodality.