Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation

Yangneng Chen, Junlin Li, Xilai Ma, Guodong DU, Wenya Wang, Min Zhang, Jing Li

Keywords: Large Vision-Language Models, Hallucination, Alignment, Language Bias

TL;DR: We diagnose that language bias in LVLMs stems from training misalignment and introduce two simple methods, LBR and LBP, that correct this during training to improve general performance and reduce hallucinations without extra model and data.

Abstract: Large Vision-Language Models (LVLMs) extend large language models with visual understanding, but remain vulnerable to hallucination, where outputs are fluent yet inconsistent with images. Recent studies link this issue to *language bias*—the tendency of LVLMs to over-rely on text while neglecting visual inputs. Yet most analyses remain empirical without uncovering its underlying cause. In this paper, we provide a systematic study of language bias and identify its root in modality misalignment during training. Our analysis shows that both Visual Instruction Tuning (VIT) and Direct Preference Optimization (DPO) often prioritize textual improvements, which may cause LVLMs to overly lean toward language modeling rather than balanced multimodal understanding. To address this, we propose two simple yet effective methods: **Language Bias Regularization (LBR)**, which mitigates language bias through regularization during instruction tuning, and **Language Bias Penalty (LBP)**, which penalizes language bias in the DPO training process. Extensive experiments across diverse models and benchmarks demonstrate the effectiveness of our approach. LBR consistently improves performance on over ten general benchmarks, while LBP significantly reduces hallucination and improves trustworthiness. Together, these methods not only mitigate language bias but also advance the overall alignment of LVLMs, all without introducing any additional data or auxiliary models.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 2884