Elicitation and Assessment of Hand-Based Gestures for Information Processing in Immersive Environments

Abstract

Objective This study aims to investigate how information processing tasks can be performed by hand-based gestures in immersive environments and to identify usable gesture designs. Background With advances in display and interaction technologies, immersive environments have evolved to support productivity-oriented activities, in which information processing plays a critical role. However, performing such tasks in immersive environments remains challenging and calls for intuitive interaction techniques aligned with users’ cognitive processes. Gestures offer a promising approach by allowing users to externalize and manipulate information through bodily actions. Methods Using 19 cognitive processes outlined in the revised Bloom’s taxonomy as representative information processing tasks, a gesture elicitation study with 15 participants explored how these tasks could be mapped to hand-based gestures (Study I). The resulting candidate gestures were subsequently validated and evaluated with 20 participants using a combination of objective and subjective measures (Study II). Results Study I produced an initial set of 32 candidate gestures, which were validated and evaluated in Study II. Four recurring mapping strategies were identified: linguistic-symbolic, spatial-manipulative, metaphoric, and social-conventional mappings. A consolidation model was then introduced to identify a recommended set of 20 gestures based on their performance across multi-dimensional measures. Conclusions Information processing tasks can be translated into usable hand-based gestures. Future gesture design should treat information as an interactive entity while carefully considering ergonomics, semantic expressiveness, and gesture similarity to improve usability. Applications The recommended gestures, mapping strategies, and design considerations can inform the gesture design for immersive systems that involve information processing tasks.