Go to RSS 2025 Workshop OOD homepageDEMONSTRATE: Zero-shot Language to Robotic Control via Multi-task Demonstration Learning
Keywords: Large Language Models, Robustness to Hallucinations, Prompt Sensitivity, Multitask Learning, Inverse Reinforcement Learning, Robotics Manipulation Tasks
TL;DR: Learn language to control from demonstrations with safety check for hallucinations before task execution.
Abstract: The integration of large language models (LLMs) with control systems has demonstrated significant potential in various settings, such as task completion with a robotic manipulator. A main reason for this success is the ability of LLMs to perform in-context learning, which, however, strongly relies on the design of task examples, closely related to the target tasks. Consequently, employing LLMs to formulate optimal control problems often requires task examples that contain explicit mathematical expressions, designed by trained engineers. Furthermore, there is often no principled way to evaluate for hallucination before task execution. To address these challenges, we propose DEMONSTRATE, a novel methodology that avoids the use of LLMs for complex optimization problem generations, and instead only relies on the embedding representations of task descriptions. To do this, we leverage tools from inverse optimal control to replace in-context prompt examples with task demonstrations, as well as the concept of multitask learning, which ensures target and example task similarity by construction. Given that hardware demonstrations can easily be collected using teleoperation or guidance of the robot, our approach reduces the reliance on engineering expertise for designing in-context examples. Furthermore, the enforced multitask structure enables learning from few demonstrations and assessment of hallucinations prior to task execution. We demonstrate the effectiveness of our method through simulation and hardware experiments involving a robotic arm tasked with tabletop manipulation.
Submission Number: 40
Loading