3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding

Ting Huang, Zeyu Zhang, Hao Tang

Keywords: 3D Scene Understanding, 3D Computer Vision, Reinforcement Learning

Abstract: Large vision-language models (VLMs) have made significant strides in 2D visual understanding tasks, sparking interest in extending these capabilities to 3D scene understanding. However, current 3D VLMs often struggle with robust reasoning and generalization due to limitations in high-quality spatial data and the static nature of viewpoint assumptions. To address these challenges, we propose 3D-R1, a general framework that enhances the reasoning capabilities of 3D VLMs. Specifically, we first construct a high-quality synthetic dataset with CoT, named Scene-30K, leveraging existing 3D-VL datasets and a data engine based on DeepSeek-R1. It serves as cold-start initialization data for 3D-R1. Moreover, we leveraging RLHF policy such as GRPO in the reinforcement learning training process to enhance reasoning capabilities and introduce two reward functions: a perception reward and a semantic similarity reward to maintain detection accuracy and answer semantic precision. Furthermore, we introduce a dynamic view selection strategy that adaptively chooses the most informative perspectives for 3D scene understanding. Extensive experiments demonstrate that 3D-R1 delivers an average improvement of 10% across various 3D scene benchmarks, highlighting its effectiveness in enhancing reasoning and generalization in 3D scene understanding.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 6132