Improved Visual-Spatial Reasoning via R1-Zero-Like Training

Zhenyi Liao, Qingsong Xie, Yanhao Zhang, Zijian Kong, Changwang Zhang, Haonan Lu, Zhijie Deng

Keywords: Spatial Reasoning, Multimodel Large Language Models, Group Relative Policy Optimization

TL;DR: We construct VSI-100k, a training dataset for spatial understanding and reasoning, to enhance the spatial abilities of multimodal large language models through various training methods and provide a detailed comparative analysis of these methods.

Abstract: Increasing attention has been placed on improving the reasoning capacities of multi-modal large language models (MLLMs). As the cornerstone for AI agents that function in the physical realm, video-based visual-spatial intelligence (VSI) emerges as one of the most pivotal reasoning capabilities of MLLMs. This work conducts a first, in-depth study on improving the visual-spatial reasoning of MLLMs via R1-Zero-like training. Technically, we first identify that the visual-spatial reasoning capacities of small- to medium-sized Qwen2-VL models cannot be activated via Chain of Thought (CoT) prompts. We then incorporate Group Relative Policy Optimization (GRPO) training for improved visual-spatial reasoning, using the carefully curated VSI-100k dataset, following DeepSeek-R1-Zero. During the investigation, we identify the necessity to keep the KL penalty (even with a small value) in GRPO. With just 120 GPU hours, our vsGRPO-2B model, fine-tuned from Qwen2-VL-2B, can outperform the base model by 12.1% and surpass GPT-4o. Moreover, our vsGRPO-7B model, fine-tuned from Qwen2-VL-7B, achieves performance comparable to that of the best open-source model LLaVA-NeXT-Video-72B. Additionally, we compare vsGRPO with supervised fine-tuning and direct preference optimization baselines in terms of both spatial and general abilities. Our observations indicate that GRPO training achieves significant performance superiority while effectively balancing spatial and general capabilities.

Submission Number: 7