Semantic-aware decoding of covert inner speech: A multimodal EEG–EMG–audio framework

Abstract

Non-invasive brain–computer interfaces (BCIs) aim to restore communication by decoding intended messages directly from neural activity, even without audible speech. However, evidence remains limited that non-invasive signals can support subject-held-out decoding of predefined semantic intentions from covert inner speech. We investigate whether a semantic-aware multimodal framework can generalize from overt spoken commands to covert inner speech using scalp electroencephalography (EEG), auxiliary electromyography (EMG), and overt-speech audio supervision. Ten healthy participants produced four everyday commands ( water , toilet , light , pain ) in overt and covert phases. Overt trials included synchronized EEG, EMG, and audio, whereas covert trials were evaluated without audio using EEG with auxiliary EMG. The proposed model learns a shared latent representation from overt multimodal supervision and covert training trials from non-held-out subjects, combining supervised contrastive learning, ArcFace classification, semantic prototype alignment, and overt–covert regularization. Evaluation follows a subject-held-out covert protocol with target-subject overt calibration: covert trials from the held-out participant are never used for training, model selection, or calibration. Across subjects, the model achieves a mean overt-validation accuracy of 0.54 and a mean covert-test accuracy of 0.42 on the four-class task, above the 0.25 chance level. Latent-space and semantic-retrieval analyses indicate that the learned representations retain class-related structure and alignment with text-based semantic prototypes. Ablations reveal that multimodal overt information supports within-subject decoding and representational analysis, whereas EEG-only overt supervision is more robust under held-out-subject covert transfer. These results identify subject variability, modality transfer, and calibration as key challenges for future semantic BCIs.