LEMON: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding

Yongyuan Liang, Furong Huang

Keywords: Large Multimodal Model, Spatial Understanding and Reasoning

Abstract:

Scaling large multimodal models (LMMs) to 3D understanding poses unique challenges: point cloud data is sparse and irregular, existing models rely on fragmented architectures with modality-specific encoders, and training pipelines often suffer from instability and poor scalability.
We introduce **LEMON**, a unified transformer architecture that addresses these challenges by jointly processing 3D point cloud patches and language tokens as a single sequence.
Unlike prior work that relies on modality-specific encoders and cross-modal alignment modules, this design enables early spatial-linguistic fusion, eliminates redundant encoders,
improves parameter efficiency, and supports more effective model scaling.
To handle the complexity of 3D data, we develop a structured patchification and tokenization scheme that preserves spatial context, and a three-stage training curriculum that progressively builds capabilities from object-level recognition to scene-level spatial reasoning.
**LEMON** establishes new state-of-the-art performance across comprehensive 3D understanding and reasoning tasks, from object recognition and captioning to spatial reasoning in 3D scenes, while demonstrating robust scaling properties as model size and training data increase. Our work provides a unified foundation for advancing 3D spatial intelligence in real-world applications.

Submission Number: 172