3D Question Answering for City Scene Understanding

Penglei Sun, Yaoxian Song, Xiang Liu, Xiaofei Yang, Qiang Wang, tiefeng li, Yang YANG, Xiaowen Chu

Abstract:

3D multimodal question answering (MQA) plays a crucial role in scene understanding by enabling intelligent agents to comprehend their surroundings in 3D environments.
While existing research has primarily focused on indoor household tasks and outdoor roadside autonomous driving tasks, there has been limited exploration of city-level scene understanding tasks.
Furthermore, existing research faces challenges in understanding city scenes, due to the absence of spatial semantic information and human-environment interaction information at the city level.
To address these challenges, we investigate 3D MQA from both dataset and method perspectives.
From the dataset perspective, we introduce a novel 3D MQA dataset named City-3DQA for city-level scene understanding,
which is the first dataset to incorporate scene semantic and human-environment interactive tasks within the city.
From the method perspective, we propose a Scene graph enhanced City-level Understanding method (Sg-CityU), which utilizes the scene graph to introduce the spatial semantic.
A new benchmark is reported and our proposed Sg-CityU achieves accuracy of 63.94 % and 63.76 % in different settings of City-3DQA.
Compared to indoor 3D MQA methods and zero-shot using advanced large language models (LLMs), Sg-CityU demonstrates state-of-the-art (SOTA) performance in robustness and generalization.
Our dataset and code are available on our project website\footnote{\url{https://sites.google.com/view/city3dqa/}}.

Primary Subject Area: [Content] Vision and Language

Secondary Subject Area: [Content] Vision and Language

Relevance To Conference: In this paper, we introduce a new 3D multimodal question answering dataset called City-3DQA for city scene understanding. Additionally, we propose a baseline method on City-3DQA named Sg-CityU.

Submission Number: 2252