Exploring the Synergetic and Divergent Potentials of Multimodal Semantics for Feature Fusion-based Video Recommendation

Abstract

Traditional multimodal video recommendation systems typically focus on semantic synergy across modalities (e.g., text, image, video) by employing attention mechanisms to weight and aggregate features, while overlooking the potential benefits of semantic divergence among modalities, which is often treated as noise. Moreover, in grid-based video recommendation platforms, existing methods fuse implicit video content into video encoders without explicit user visibility, leading to factually inconsistent representations. To address these limitations, we propose ESDvr (E xploring the S ynergetic and D ivergent Potentials of Multimodal Semantics for Feature Fusion-based V ideo R ecommendation ), a novel framework that jointly models cross-modal semantic divergence and synergy. Specifically, we introduce Mixture of Gaussian Blur Gating (MoGG) to capture semantic discrepancies between cover images and texts while integrating them with synergistic features. Additionally, based on cross-modal semantic synergy, we design a learnable threshold-based mechanism to dynamically simulate user hover behaviors, adaptively enhancing video encoding with content tailored to individual users. For user encoding, we leverage recently watched videos to model personalized interests. Furthermore, a contrastive loss is proposed to align video features derived from multimodal semantics with those from user-item interaction histories, ensuring semantic consistency. Rich experiments on MicroLens and MovieLens datasets demonstrate that ESDvr outperforms state-of-the-art (SOTA) baselines, achieving significant improvements of 10.15% (Recall@5) , 10.22% (NDCG@10) .