Domain-Sensitive Patterns in Student Writing: Comparing Human, AI-Revised, and AI-Generated Abstracts in Human Factors

Abstract

Generative AI is increasingly used in academic writing, although many studies compare only human-written and AI-generated text. This within-subjects study examined manual, AI-revised, and AI-generated extended abstracts in a graduate human factors course. Text analyses included 96 abstracts from 32 participants, and repeated-measures survey analyses included 30 participants. We compared structural and lexical features, readability, Sentence-BERT embeddings, exploratory writing-condition classification, technology acceptance, and perceived workload. Manual abstracts showed the greatest variability. AI-generated abstracts showed less structural variability, higher lexical density, and greater estimated reading difficulty. AI-revised abstracts showed metric-specific overlap with both other conditions. A three-class classifier yielded 68.75% accuracy under abstract-level leave-one-out cross-validation, although participant and topic overlap limits generalization. Participants reported greater acceptance and lower workload in the AI-supported conditions. These preliminary findings motivate examining degrees of AI involvement and using participant-grouped validation in future studies.