From Entailment to Contradiction: Confidence Calibration in MLLMs via Difficulty-Guided Optimization

ACL ARR 2026 January Submission1489 Authors

Keywords: Calibration, LLM, Reinforcement Learning

Abstract: The ability to accurately assess and calibrate the confidence of large language models (LLMs) is critical for improving their performance in real-world applications. While much progress has been made in confidence calibration for text-only models, the challenge remains underdeveloped in multimodal large language models (MLLMs), which often suffer from hallucinations and poorly calibrated confidence due to factors such as visual ambiguity and the presence of rare entities. This research aims to tackle the problem of confidence miscalibration in MLLMs by proposing a novel two-stage framework. The first stage focuses on analyzing dataset characteristics, such as contradiction rates and entity rarity, which contribute to task difficulty. The second stage involves fine-tuning the model to better estimate confidence and reduce hallucinations. Our approach improves the reliability of confidence estimates and significantly reduces hallucination rates, offering a step forward in developing more trustworthy multimodal models.

Paper Type: Long

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: Interpretability and Analysis of Models for NLP, Special Theme Track

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 1489