RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs

Sangmin Woo, Jaehyuk Jang, Donguk Kim, Yubin Choi, Changick Kim

Keywords: Large Vision Language Models

TL;DR: Random Image Transformations can reduce Hallucinations in Large Vision Language Models

Abstract: Recent advancements in Large Vision Language Models (LVLMs) have revolutionized how machines understand and generate textual responses based on visual inputs. Despite their impressive capabilities, they often produce "hallucinatory" outputs that do not accurately reflect the visual information, posing challenges in reliability and trustworthiness. Inspired by test-time augmentation, we propose a simple, training-free method termed RITUAL to enhance robustness against hallucinations in LVLMs. RITUAL introduces random image transformations as complementary inputs during the decoding phase. Importantly, these transformations are not employed during the training of the LVLMs. This straightforward strategy reduces the likelihood of hallucinations by exposing the model to varied visual scenarios, enriching its decision-making process. While transformed images alone may initially degrade performance, we empirically find that strategically combining them with the original images mitigates hallucinations. Specifically, in cases where hallucinations occur with the original image, the transformed images help correct misinterpretations by adjusting the probability distribution. By diversifying the visual input space, RITUAL provides a more robust foundation for generating accurate outputs. Notably, our method works seamlessly with existing contrastive decoding methods and does not require external models or costly self-feedback mechanisms, making it a practical addition. While extremely simple, RITUAL significantly outperforms existing contrastive decoding methods across several object hallucination benchmarks, including POPE, CHAIR, and MME.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9078