The Validity and Reliability of MUETBot in Supporting the Learning of MUET Reading Skills among ESL Students

Abstract

HRMARS - The Malaysian University English Test (MUET) presents persistent challenges for learners of English as a second language (ESL), particularly in the reading component, which requires them to comprehend complex academic texts under considerable time pressure. To address this challenge, MUETBot, a conversational tool supported by artificial intelligence (AI), was designed and developed to facilitate independent learning of MUET reading skills. This paper reports the evaluation phase within the Design and Development Research (DDR) framework, focusing on content validity, user acceptance, and the reliability of the evaluation instrument. Five subject matter experts assessed content validity using a structured validation form, from which item and scale level content validity indices were calculated. User acceptance was subsequently evaluated through a User Acceptance Test (UAT) administered to 30 undergraduate ESL students on two occasions separated by two weeks. The expert evaluation produced an average scale level content validity index of .92, indicating strong alignment between the MUETBot content and the constructs underlying MUET reading. The UAT scores demonstrated good internal consistency (Cronbach’s ? = .824) and strong temporal stability (r = .85, p < .001). Students also reported high levels of acceptance across all five constructs, with mean scores ranging from 4.33 to 4.47 on a five point scale. The findings indicate that MUETBot provides content aligned with the MUET reading construct and is positively accepted by its intended users. The study also presents an empirically grounded approach to validating language learning tools supported by AI before examining their effects on learning outcomes.