Person-independent and cross-dataset evaluation of convolutional and recurrent architectures for eye movement classification

Abstract

Abstract Classifying eye movements into fixations, saccades, and smooth pursuits is a basic step in the analysis of gaze recordings. Deep models are now common for this task, but they are often evaluated on windows that share participants between training and test sets, which leaves open how well they work for people the model has not seen. This paper reports a person-independent comparison of three architectures—a one-dimensional CNN, an LSTM, and a CNN–LSTM hybrid—for four-class eye movement classification on the GazeCom dataset (54 participants, 250 Hz, expert frame-level annotations). Each model is trained and tested under leave-one-person-out cross-validation, so every participant is held out in turn, and each architecture is run with two input representations: raw gaze position and a derived set that adds velocity, acceleration, and angular change. Model differences are assessed at the participant level (paired tests over 54 folds, Holm-corrected) rather than on overlapping windows. Two findings are consistent across the study. First, recurrent modeling is what separates the strong models from the weak one: the LSTM and the hybrid reach a macro-F1 of about 0.82 with the derived input, while the CNN reaches 0.71, and the gap holds for every participant. Second, adding the convolutional front end to the LSTM does not help: the hybrid and the plain LSTM are statistically indistinguishable with the raw input and separated by less than 0.01 macro-F1 with the derived input, so the extra component is not warranted by the data; the hybrid carries about 70% more parameters than the LSTM for no measurable gain. The derived input improves every architecture, most for the CNN and for the smooth-pursuit class. Three non-learned or lightweight baselines (velocity threshold, velocity–dispersion, and a Random Forest on kinematic features), tuned per fold and evaluated under the same protocol, stay near 0.53–0.54 macro-F1 on the fixation/saccade/pursuit task, well below all three deep models. When the GazeCom-trained models are tested without retraining on the Lund2013 recordings and scored against both expert raters, accuracy falls as expected, but the ordering of the architectures is preserved. The plain LSTM is therefore an adequate model for this task; the smooth-pursuit class remains the main source of error and is the most useful target for future work.