Towards Omnidirectional Reasoning: A Dataset, Benchmark, and GRPO-based Method

ICLR 2026 Conference Submission24458 Authors

Keywords: Multi-Modal Large Models, Omnidirectional Vision

Abstract: Omnidirectional images (ODIs), with their 360° field of view, provide unparalleled spatial awareness for immersive applications like augmented reality and embodied AI. However, the capability of existing multimodal large language models (MLLMs) to comprehend and reason about such panoramic scenes remains underexplored. This paper addresses this gap by introducing OmniVQA, the \textbf{\textit{first}} dataset and conducting the \textbf{\textit{first}} benchmark for omnidirectional visual question answering. Our evaluation of state-of-the-art MLLMs reveals significant limitations in handling omnidirectional visual question answering, highlighting persistent challenges in object localization, feature extraction, and hallucination suppression within panoramic contexts. These results underscore the disconnect between current MLLM capabilities and the demands of omnidirectional visual understanding, which calls for dedicated architectural or training innovations tailored to 360 ° imagery.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 24458