Implicit and Explicit Visual Information Aware Multimodal Relation Extraction

Abstract

Recently, Multimodal Relation Extraction (MRE) has achieved great improvements. The inherent abstraction and variability of visual content make it challenging to extract relational information in multimodal relation extraction. Beyond identifying explicit visual entities for filtering out noisy information, it is also essential to capture implicit visual background knowledge that aids relation reasoning. Collectively, these factors are critical in alleviating the error sensitivity issues caused by irrelevant visual information. In light of this, we propose an Implicit and Explicit Visual Information Aware Network (IEVAN) for Multimodal Relation Extraction. Specifically, the Caption-Aware Module (CAM) hierarchically aligns the fine-grained visual entities and textual entities for reasoning. The Knowledge-Aware Module (KAM) supplements relation extraction with background knowledge. In addition, for preserving crucial information within different modalities, we leverage a Mutual Information Maximization method to regulate the multimodal reasoning modules. Experiments show that our model outperforms the state-of-the-art MRE models on the benchmark dataset MNRE and MORE. Further ablation studies prove the effective performance of our methods.