Tell Me Where to Go: A Composable Framework for Context-Aware Embodied Robot Navigation

Harel Biggie, Ajay Narasimha Mopidevi, Dusty Woods, Chris Heckman

Keywords: Natural language, navigation, contextual navigation

TL;DR: A composable framework for mobile robot navigation using modular blocks for natural language parsing, image understanding, grounding, and path planning in unknown environments.

Abstract: Humans have the remarkable ability to navigate through unfamiliar environments by solely relying on our prior knowledge and descriptions of the environment. For robots to perform the same type of navigation, they need to be able to associate natural language descriptions with their associated physical environment with a limited amount of prior knowledge. Recently, Large Language Models (LLMs) have been able to reason over billions of parameters and utilize them in multi-modal chat-based natural language responses. However, LLMs lack real-world awareness and their outputs are not always predictable. In this work, we develop a low-bandwidth framework that solves this lack of real-world generalization by creating an intermediate layer between an LLM and a robot navigation framework in the form of Python code. Our intermediate shoehorns the vast prior knowledge inherent in an LLM model into a series of input and output API instructions that a mobile robot can understand. We evaluate our method across four different environments and command classes on a mobile robot and highlight our framework's ability to interpret contextual commands.

Student First Author: yes

Instructions: I have read the instructions for authors (https://corl2023.org/instructions-for-authors/)

Video: https://www.youtube.com/watch?v=ZRWZpsAtGdE

Code: https://github.com/arpg/navcon