ProReason: Multi-Modal Proactive Reasoning with Decoupled Eyesight and Wisdom
Keywords: Proactive reasoning, Problem-oriented, LLM-assisted multi-modal reasoning, Large vision-language model, Visual reasoning, Multi-step reasoning framework
TL;DR: We introduce ProReason, a multi-modal reasoning framework with proactive perception and decoupled vision-reasoning capabilities, surpassing current SOTA methods and highlighting the potential for LLM-assisted visual reasoning in future research.
Abstract: Large vision-language models (LVLMs) have witnessed significant progress on visual understanding tasks.
However, they often prioritize language knowledge over image information on visual reasoning tasks, incurring performance degradation.
To tackle this issue, we first identify the drawbacks of existing solutions (i.e., insufficient and irrelevant visual descriptions, and limited multi-modal capacities).
We then decompose visual reasoning process into two stages: visual perception (i.e., eyesight) and textual reasoning (i.e., wisdom), and introduce a novel visual reasoning framework named ProReason.
This framework features multi-run proactive perception and decoupled vision-reasoning capabilities.
Briefly, given a multi-modal question, ProReason iterates
proactive information collection and reasoning
until the answer can be concluded with necessary and sufficient visual descriptions.
Notably, the disassociation of capabilities allows seamless integration of existing large language models (LLMs) to compensate for the reasoning deficits of LVLMs.
Our extensive experiments demonstrate that ProReason outperforms both existing multi-step reasoning frameworks and passive peer methods on a wide range of benchmarks
for both open-source and closed-source models.
In addition, with the assistance of LLMs,
ProReason achieves a performance improvement of up to 15\%
on MMMU benchmark.
Our insights into existing solutions and the decoupled perspective for feasible integration of LLMs illuminate future research on visual reasoning techniques, especially LLM-assisted ones.
Primary Area: applications to computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4444
Loading