Hybrid hypergraph networks for multimodal sequence data classification

Abstract

Abstract Modeling temporal multimodal data requires both long-range temporal dependencies and cross-modal interactions to be represented. Existing approaches commonly model each modality separately and then apply shallow fusion, which can limit the representation of higher-order temporal structure. We propose a hybrid hypergraph network (HHN) that combines an established order-preserving alignment for candidate cross-modal links with an entropy-guided mechanism for constructing intra-modal hyperedges. Timestamped segments are represented as nodes; within each modality, an adaptive temporal window and a maximum-entropy-difference rule form higher-order hyperedges that are aggregated by hypergraph convolution. Across modalities, temporally weighted pairwise links and graph attention provide cross-modal fusion. Across four processed multimodal datasets, the reported mean average precision and area under the receiver operating characteristic curve were higher for HHN than for the evaluated baselines. Ablation results further indicate that both the hybrid graph modules and multimodal inputs contribute to classification performance.