Hallucination-aware Intermediate Representation Editing in Large Vision-Lanugage Models

Wei Suo, Hanzu Zhang, Lijun Zhang, Ji Ma, PENG WANG, Yanning Zhang

Keywords: mitigating hallucination, feature editing, LVLMs

Abstract: Large Vision-Language Models have demonstrated exceptional performance in multimodal reasoning and complex scene understanding. However, these models still face significant hallucination issues, where outputs contradict visual facts. Recent research on hallucination mitigation has focused on retraining methods and Contrastive Decoding (CD) methods. While both methods perform well, retraining methods require substantial training resources, and CD methods introduce dual inference overhead. These factors hinder their practical applicability. To address the above issue, we propose a framework for dynamically detecting hallucination representations and performing hallucination-eliminating edits on these representations. With minimal additional computational cost, we achieve state-of-the-art performance on existing benchmarks. Extensive experiments demonstrate the effectiveness of our approach, highlighting its efficient and robust hallucination elimination capability and its powerful controllability over hallucinations. Our code will be released.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 23142