Signage-Aware Exploration in Open World using Venue Maps

Chang Chen, Liang Lu, Lei Yang, Yinqiang Zhang, CHEN YIZHOU, Ruixing Jia, Jia Pan

Keywords: Autonomous Agents, Semantic Scene Understanding, Mapping, Planning under Uncertainty

TL;DR: This paper presents a generalizable signage-aware exploration system for searching for shops or restaurants in open-world scenarios.

Abstract: Current exploration methods struggle to search for shops or restaurants in unknown open-world environments due to the lack of prior knowledge. Humans can leverage venue maps that offer valuable scene priors to aid exploration planning by correlating the signage in the scene with landmark names on the map. However, arbitrary shapes and styles of the texts on signage, along with multi-view inconsistencies, pose significant challenges for robots to recognize signage accurately. Additionally, discrepancies between real-world environments and venue maps hinder the integration of text-level information into the planners. This paper introduces a novel signage-aware exploration system to address these challenges, enabling robots to utilize venue maps effectively. We propose a signage understanding method that accurately detects and recognizes the texts on signage using a diffusion-based text instance retrieval method combined with a 2D-to-3D semantic fusion strategy. Furthermore, we design a venue map-guided exploration-exploitation planner that balances exploration in unknown regions using directional heuristics derived from venue maps and exploitation for perceiving the signage actively. Experiments in large-scale shopping malls demonstrate our method's superior signage recognition performance and search efficiency, surpassing state-of-the-art text spotting methods and traditional exploration approaches. Codes and videos are on our project website:https://sites.google.com/view/signage-aware-exploration.

Submission Number: 5