Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments – Extended Abstract

Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, Stefan Lee

Abstract: We develop a language-guided navigation task set in a continuous 3D environment where agents must execute low-level actions to follow natural language navigation directions. By being situated in continuous environments, this setting lifts a number of assumptions implicit in prior work that represents environments as a sparse graph of panoramas with edges corresponding to navigability. Specifically, our setting drops the presumptions of known environment topologies, short-range oracle navigation, and perfect agent localization. To contextualize this new task, we develop models that mirror many of the advances made in prior settings. We find significantly lower performance in the continuous setting -- suggesting that performance in topological settings may be inflated by the strong implicit assumptions.

TL;DR: We develop a language-guided navigation task set in a continuous 3D environment that drops several assumptions implicit in prior work set in topologically-represented environments.

Keywords: Vision-and-Language Navigation, Embodied Agents

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/beyond-the-nav-graph-vision-and-language/code)