A dual-domain guided emotion-specialized swin transformer for enhancing speech emotion recognition

Abstract

Existing Transformer-based speech emotion recognition methods primarily focus on capturing intra-correlations of a sequence through multi-head self-attention mechanisms, or on capturing local key emotional information using windowed attention. However, they neglect the aggregation of inter-relationships between different local regions, which constitutes important cues for speech emotional representation. To address this limitation, this paper proposes a dual-domain guided emotion-specialized Swin Transformer (DGEST), which can jointly identify emotionally salient regions in both temporal and frequency domains and establish emotional dependency interactions between regions. The proposed method first employs a heterogeneous dual-branch architecture to independently encode temporal and frequency domain features. Subsequently, it partitions time–frequency regions according to the inherent spectral distribution variations of emotional expressions. Finally, it achieves dual-domain emotional salient region focusing, and inter-regional dependency relationships through a hierarchical attention mechanism. Extensive experiments on public datasets IEMOCAP, CASIA, and EMODB demonstrate the superiority and efficiency of the proposed DGEST method. This research provides novel insights for dual-domain modeling in robust emotion perception and offers a practical and advanced technical solution for speech emotion recognition tasks.