Text2Motion: From Natural Language Instructions to Feasible Plans

Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, Jeannette Bohg

Abstract: We propose Text2Motion, a language-based planning framework enabling robots to solve sequential manipulation tasks specified from natural language instructions and require long-horizon reasoning. Given a natural language instruction, our framework constructs both a task- and policy-level plan that is verified to reach inferred symbolic goals. Text2Motion uses skill feasibility heuristics encoded in learned Q-functions to guide task planning with Large Language Models. Whereas previous language-based planners only consider the feasibility of individual skills, Text2Motion actively resolves geometric dependencies spanning skill sequences by performing policy sequence optimization during its search. We evaluate our method on a suite of problems that require long-horizon reasoning, interpretation of abstract goals, and handling of semantic partial observability. Our experiments show that Text2Motion can solve these challenging problems with a success rate of 64%, which is significantly higher than the prior best language-based planning method (13%). Text2Motion thus provides promising generalization characteristics to semantically diverse sequential manipulation tasks with geometric dependencies between actions.