Foundation Models to the Rescue: Deadlock Resolution in Connected Multi-Robot Systems
Keywords: Foundation models for robotics, multi-robot systems, hierarchical control framework
TL;DR: This work explores the possibility of using text-based models, i.e., large language models, and text-and-image-based models, i.e., vision-language models, as high-level planners for deadlock resolution in multi-robot systems.
Abstract: Connected multi-agent robotic systems (MRS) are prone to deadlocks in an obstacle environment where the robots can get stuck away from their desired locations under a smooth low-level control policy. Without an external intervention, often in terms of a high-level command, a low-level control policy cannot resolve such deadlocks. Utilizing the generalizability and low data requirements of foundation models, this paper explores the possibility of using text-based models, i.e., large language models (LLMs), and text-and-image-based models, i.e., vision-language models (VLMs), as high-level planners for deadlock resolution. We propose a hierarchical control framework where a foundation model-based high-level planner helps to resolve deadlocks by assigning a leader to the MRS along with a set of waypoints for the MRS leader. Then, a low-level distributed control policy based on graph neural networks is executed to safely follow these waypoints, thereby evading the deadlock. We conduct extensive experiments on various MRS environments using the best available pre-trained LLMs and VLMs.
Submission Number: 4
Loading