Sentiment Confidence Separation: A Trust-Optimized Framework for Multimodal Sentiment Classification

Zemin Tang, Min Shi, Zhibang Yang, Xu Zhou, Cen Chen, Joey Tianyi Zhou

Abstract: The Multimodal Sentiment Classification (MSC) task aims to discern sentiments from diverse data sources. Existing efforts focus on integrating multimodal features and enhancing representation learning for improved recognition. The widespread use of MSC, particularly in risk-associated domains, highlights the need for heightened trustworthiness in predictions. However, most current MSC models often provide elevated confidence regardless of whether the prediction is correct or not, with less emphasis on whether this confidence reasonably reflects the model’s certainty in predictions. This paper proposes a novel confidence optimization framework, Sentiment Confidence Separation (SCS), which helps address unreliability in MSC models by making the correct and incorrect predictions output discriminative confidences. SCS comprises Confidence Separation Loss (CSL) and Flatness-Based Separation Optimization (FBSO), facilitating reliable and precise predictions. Comprehensive experimentation validates the efficacy of the proposed approach across multiple mainstream datasets.