Multiview Scene Graph

Juexiao Zhang, Gao Zhu, Sihang Li, Xinhao Liu, Haorui Song, Xinran Tang, Chen Feng

Keywords: scene representation; spatial understanding; place recognition; object correspondence; scene graph

TL;DR: building a place+object multiview scene graph from unposed images as a topological scene representation

Abstract:

A proper scene representation is central to the pursuit of spatial intelligence where agents can robustly reconstruct and efficiently understand 3D scenes. A scene representation is either metric, such as landmark maps in 3D reconstruction, 3D bounding boxes in object detection, or voxel grids in occupancy prediction, or topological, such as pose graphs with loop closures in SLAM or visibility graphs in SfM.
In this work, we propose to build Multiview Scene Graphs (MSG) from unposed images, representing a scene topologically with interconnected place and object nodes.
The task of building MSG is challenging for existing representation learning methods since it needs to jointly address both visual place recognition, object detection, and object association from images with limited fields of view and potentially large viewpoint changes.
To evaluate any method tackling this task, we developed an MSG dataset and annotation based on a public 3D dataset.
We also propose an evaluation metric based on the intersection-over-union score of MSG edges.
Moreover, we develop a novel baseline method built on mainstream pretrained vision models, combining visual place recognition and object association into one Transformer decoder architecture.
Experiments demonstrate that our method has superior performance compared to existing relevant baselines.

Primary Area: Machine vision

Submission Number: 18923