Leveraging Weak Cross-Modal Guidance for Coherence Modelling via Iterative Learning
Abstract: Cross-modal coherence modeling is essential for intelligent systems to help them organize and structure information, thereby understanding and creating content of the physical world coherently like human-beings. Previous work on cross-modal coherence modeling attempted to leverage the order information from another modality to assist the coherence recovering of the target modality. Despite of the effectiveness, labeled associated coherency information is not always available and might be costly to acquire, making the cross-modal guidance hard to leverage. To tackle this challenge, this paper explores a new way to take advantage of cross-modal guidance without gold labels on coherency, and proposes the Weak Cross-Modal Guided Ordering (WeGO) model. More specifically, it leverages high-confidence predicted pairwise order in one modality as reference information to guide the coherence modeling in another. An iterative learning paradigm is further designed to jointly optimize the coherence modeling in two modalities with selected guidance from each other. The iterative cross-modal boosting also functions in inference to further enhance coherence prediction in each modality. Experimental results on two public datasets have demonstrated that the proposed method outperforms existing methods for cross-modal coherence modeling tasks. Major technical modules have been evaluated effective through ablation studies. \textcolor{blue}{Codes are available at: \textit{\url{https://github.com/scvready123/IterWeGO}}}.
Primary Subject Area: [Content] Multimodal Fusion
Relevance To Conference: This work aims to improve coherence modelling with weak cross-modal guidance, which is a classic and intrisinc problem in multimodal processing. In this work, we propose to leverage the weak cross-modal coherence guidance via a cross-modal relative order alignment module, then pass the message across different modalities iteratively. The coherence recovering performance could be significant improved with our method, which indicates consistant contribution to this community.
Submission Number: 5528
Loading