KHaR: Knowledge Harvesting and Refinement for multimodal sentiment analysis

ACL ARR 2025 May Submission2017 Authors

Abstract: Multimodal sentiment analysis deduces a user's sentiment by integrating information from different modalities. Previous methods mainly focus on using complex fusion networks to learn effective joint embeddings and achieved significant improvement, while ignoring refined processing within modalities, resulting in models that achieve suboptimal results on more delicate sentiment analysis. In this paper, we propose a novel framework KHaR, which harvests rich intra-modality knowledge through domain-specific adapters and utilizes mixture of experts to refine the knowledge to capture more detailed intra-modality information. In addition, we design a contrastive learning to further explore the information correlation between samples of similar sentiment intensity. To fuse the extracted features effectively, we employ the multimodal information bottleneck to filter out irrelevant information and retain the most salient features for sentiment analysis. Extensive experiments show that KHaR achieves superior performance on four benchmark datasets, and especially achieves significant improvement on the more refined sentiment analysis(e.g.Acc-7, Acc-5).

Paper Type: Long

Research Area: Sentiment Analysis, Stylistic Analysis, and Argument Mining

Research Area Keywords: style analysis,applications

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English, Chinese

Submission Number: 2017