Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning

TMLR Paper8110 Authors

Abstract: Learning general-purpose reasoning capabilities has long been a challenging problem in AI. Recent research in LLMs, such as DeepSeek-R1, has shown that reinforcement learning techniques like GRPO enable pre-trained LLMs to develop reasoning capabilities using simple question-answer pairs. In this paper, we aim to train visual language models (VLMs) to perform reasoning on image data through reinforcement learning and visual question-answer pairs, without explicitly using any chain-of-thought (CoT) supervision. Our key finding indicates that simply applying GRPO to a VLM---by prompting the model to think step by step before giving an answer---may cause the model to develop shortcuts from easy questions, resulting in poor generalization of reasoning to broader question domains. We argue that the key to mitigating shortcut learning is to encourage the model to interpret images prior to reasoning. Therefore, we train the model to adhere to a caption-reason-answer output format: initially generating a detailed caption for an image, followed by constructing an extensive reasoning chain. When trained on 273K CoT-free visual question-answer pairs and using only reinforcement learning, our model, named Visionary-R1, outperforms strong multimodal models (e.g., GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro) on multiple visual reasoning benchmarks. Code and models will be open-sourced.

Submission Type: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Qi_Yu1

Submission Number: 8110