Towards Structure-aware Model for Multi-modal Knowledge Graph Completion

Linyu Li

Abstract:

Knowledge graphs (KGs) play a key role in promoting various multimedia and AI applications. However, with
the explosive growth of multi-modal information, traditional
knowledge graph completion (KGC) models cannot be directly
applied. This has attracted a large number of researchers to
study multi-modal knowledge graph completion (MMKGC).
Since MMKG extends KG to the visual and textual domains,
MMKGC faces two main challenges: (1) how to deal with the
fine-grained modality information interaction and awareness;
(2) how to ensure the dominant role of graph structure in
multi-modal knowledge fusion and deal with the noise generated
by other modalities during modality fusion. To address these
challenges, this paper proposes a novel MMKGC model named
TSAM, which integrates fine-grained modality interaction and
dominant graph structure to form a high-performance MMKGC
framework. Specifically, to solve the challenges, TSAM proposes
the Fine-grained Modality Awareness Fusion method (FgMAF),
which uses pre-trained language models to better capture finegrained semantic information interaction of different modalities
and employs an attention mechanism to achieve fine-grained
modality awareness and fusion. Additionally, TSAM presents the
Structure-aware Contrastive Learning method (SaCL), which utilizes two contrastive learning approaches to align other modalities
more closely with the structured modality. Extensive experiments
show that the proposed TSAM model significantly outperforms
existing MMKGC models on widely used multi-modal datasets.