Robotic Information Gathering using Semantic Language Instructions

Ian C. Rankin, Seth McCammon, Geoffrey A. Hollinger

Abstract: This paper presents a framework that uses language instructions to define the constraints and objectives for robots gathering information about their environment. Designing autonomous robotic sampling missions requires deep knowledge of both autonomy systems and scientific domain expertise. Language commands provide an intuitive interface for operators to give complex instructions to robots. The key insight we leverage is using topological constraints to define routing directions from the language instruction such as ‘route to the left of the island.’ This work introduces three main contributions: a framework to map language instructions to constraints and rewards for robot planners, a topology constrained information gathering algorithm, and an automatic semantic feature detection algorithm for upwelling fronts. Our work improves on existing methods by not requiring training data with language instruction to planner constraint pairs, allowing new robotic domains such as marine robotics to use our method. This paper provides results demonstrating our framework producing correct constraints for 84.6% of instructions, from a systematically generated corpus of over 1.1 million instructions We also demonstrate the framework producing robot plans from language instructions for real-world scientific sampling missions with the Slocum underwater glider.