Enhancing safety of vision-language reasoning through model-to-model deliberation

SUNGWOO KIM, Yongjin Lee, Yunsick Sung

Abstract: Traditional vision-language models demonstrate strong performance in tasks such as image captioning and visual question answering, but they remain limited by issues such as hallucination, lack of self-correction, and shallow reasoning. These shortcomings compromise the safety, robustness, and consistency of their reasoning, particularly in ambiguous or high-stakes scenarios. In this paper, we propose three complementary frameworks aimed at enabling more trustworthy visual reasoning through structured deliberation. The first is the Self-Reflective Reasoning Single-Agent Framework, which facilitates iterative self-revision without requiring external supervision. The second is the Structured Debate Agent Framework, in which turn-based rebuttals between agents promote contrastive, multi-perspective refinement. The third is the Progressive Two-Stage Debate Agent Framework, which enables efficient yet accurate decision-making through model-to-model deliberation between smaller and larger agents. Experiments on the COCO dataset demonstrate that all three frameworks significantly enhance reasoning performance, achieving up to a 5.4\% improvement in intersection over union (IoU) and over a 40\% reduction in localization error compared to a single-pass baseline. Further evaluation across robustness (IoU), safety (self-revision rate, SRR), and consistency (consistency score, CS) confirms the effectiveness of multi-round, self-corrective, and multi-agent reasoning strategies. These results establish a practical path toward safer, more robust, and more interpretable vision-language models through lightweight, deliberative inference frameworks.