RoboHop: Segment-based Topological Map Representation for Open-World Visual Navigation

Sourav Garg, Krishan Rana, Mehdi Hosseinzadeh, Lachlan Mares, Niko Suenderhauf, Feras Dayoub, Ian Reid

Keywords: Topological, Mapping, Planning, Navigation, Zero-shot, Open-Vocabulary, Segmentation

TL;DR: A segment-based topological map representation to generate navigation plans from open-vocabulary queries in the form of `hops' over segments to reach the target goal, without the need for a learned policy.

Abstract:

Mapping is crucial for spatial reasoning, planning and robot navigation. Existing approaches range from metric, which require precise geometry-based optimization, to purely topological, where image-as-node based graphs lack explicit object-level reasoning and interconnectivity. In this paper, we propose a novel topological representation of an environment based on *image segments*, which are semantically meaningful and open-vocabulary queryable, conferring several advantages over previous works based on pixel-level features. Unlike 3D scene graphs, we create a purely topological graph with segments as nodes, where edges are formed by **a)** associating segment-level descriptors between pairs of consecutive images and **b)** connecting neighboring segments within an image using their pixel centroids. This unveils a *continuous sense of a place*, defined by inter-image persistence of segments along with their intra-image neighbours. It further enables us to represent and update segment-level descriptors through neighborhood aggregation using graph convolution layers, which improves robot localization based on segment-level retrieval.
Using real-world data, we show how our proposed map representation can be used to **i)** generate navigation plans in the form of *hops over segments* and **ii)** search for target objects using natural language queries describing spatial relations of objects. Furthermore, we quantitatively analyze data association at the segment level, which underpins inter-image connectivity during mapping and segment-level localization when revisiting the same place. Finally, we show preliminary trials on segment-level *hopping* based zero-shot real-world navigation. Supplementary details can be found on our project page: oravus.github.io/RoboHop/

Submission Number: 48