SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning in Text-Only LLMs

ICLR 2026 Conference Submission22664 Authors

Keywords: Multimodal Reasoning, Multi-agent LLM, Visual Question Answering, Tool Use, Vision-Language Model

TL;DR: A small VLM agent “translates” VQA-tasks into structured intermediate representations (SIRs) for text-only LLMs, enabling agentic multimodal reasoning that beats larger end-to-end VLMs at lower cost.

Abstract:

Recent advances in text-only large language models (LLMs), such as DeepSeek-R1, demonstrate remarkable reasoning ability. However, these models remain fragile—or entirely incapable—when extended to multimodal tasks. Existing approaches largely rely on single-form captions, which lack diversity and often fail to adapt across different types of Visual Question Answering (VQA) benchmarks. As a result, they provide no principled or efficient channel for transmitting fine-grained visual information.
We introduce **SeeingEye**, a modular framework that unlocks multimodal reasoning in text-only LLMs through an **agent-based small VLM translator**. This translator acts as a perception agent: it can invoke specialized tools (e.g., OCR and crop) and iteratively distill multimodal inputs into structured intermediate representations (SIRs) tailored to the question. These SIRs are then passed to the text-only LLM, which serves as a reasoning agent. Crucially, the translator and reasoner engage in multi-round feedback and interaction, enabling the extraction of targeted visual details and yielding more confident answers.
Experiments on knowledge-intensive VQA benchmarks, including MMMU and MIA-Bench, demonstrate that Translation First not only reduces inference cost but also surpasses much larger end-to-end VLMs. For example, an instantiation combining a 3B-parameter vision translator with an 8B-parameter language reasoner outperforms a monolithic 32B VLM on challenging knowledge-based questions.
Our results highlight that decoupling perception from reasoning via agent information flow offers a scalable and plug-and-play pathway to multimodal reasoning, allowing strong text-only LLMs to fully leverage their reasoning capabilities.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 22664