Spatial-LLaVA: Enhancing Large Language Models with Spatial Referring Expressions for Visual Understanding
Workshop Statement: Our paper focus on the topic of efficiently fine-tuning large multimodal models for embodied AI tasks that require spatial understanding. We fine-tuned LLaVA on our newly introduced SUN-Spot v2.0 dataset, which contains RGB-D images along with spatial referring expressions and detailed object annotations. To guide the model's learning, we used Set-of-Marks prompting, a technique that explicitly links objects in the image to words in the caption. This helped establish a stronger alignment between visual objects and spatial language, enabling the model to better understand and reason about spatial relationships. This work fits closely with the workshop’s theme of making large models more adaptable to real-world, human-centered robotics—especially in scenarios where understanding spatial language is critical for safe and effective interaction.
Keywords: computer vision; human robot interaction; natural language
Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable abilities in comprehending visual input alongside text input. Typically, these models are trained on extensive data sourced from the internet, which are sufficient for general tasks such as scene understanding and question answering. However, they often underperform on specialized tasks where online data is scarce, such as determining spatial relationships between objects or localizing unique target objects within a group of objects sharing similar features. In response to this challenge, we introduce the SUN-Spot v2.0 dataset, now comprising a total of 90k image-caption pairs and additional annotations on the landmark objects. Each image-caption pair utilizes Set-of-Marks prompting as an additional indicator, mapping each landmark object in the image to the corresponding object mentioned in the caption. Furthermore, we present Spatial-LLaVA, an MLLM trained on conversational data generated by a state-of-the-art language model using the SUN-Spot v2.0 dataset. Our approach ensures a robust alignment between the objects in the images and their corresponding object mentions in the captions, enabling our model to learn spatial referring expressions without bias from the semantic information of the objects. Spatial-LLaVA outperforms previous methods by 3.15\% on the zero-shot Visual Spatial Reasoning benchmark dataset. Spatial-LLaVA is specifically designed to precisely understand spatial referring expressions, making it highly applicable for tasks in real-world scenarios such as autonomous navigation and interactive robotics, where precise object recognition is critical.
Submission Number: 11
Loading