Triboelectric E-Skin Patch Decoding Silent Speech for Advanced Human–Machine Interaction and Augmented Virtual Communication

Abstract

Silent speech decoding (lip reading) represents an innovative hands-free communication method for individuals with laryngeal and lingual impairments, addressing critical daily communication challenges. Herein, a novel silent speech decoding system is proposed, which employs a self-powered, low-cost, and conformable triboelectric-based E‑skin patch and a well-trained multi-layer convolutional neural network (CNN) model. We systematically characterize the structural principles and electrical performance of the E-skin patch. A comparative analysis of labial kinematics during vocalized and silent speech is also conducted, which demonstrates strong consistency in time-frequency domain characteristics. The utilized CNN model is trained across 15 lexical categories (200 samples per category), achieving a high accuracy of 98.75% and demonstrating excellent recognition performance. Furthermore, by integrating human-machine interaction (HMI) and virtual reality (VR) technologies, we develop three representative applications, including a real-time lip language visualization system, an orofacial-controlled "Feeding Frenzy" interactive game, and a VR communication platform with avatar dialogue. This technology enables barrier-free communication for voiceless populations, demonstrating significant potential in HMI, augmentative communication systems, virtual reality integration, and biomedical rehabilitation.