EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agents
Keywords: Embodied Artificial Intelligence, LLM Multi-agent System, Multi-robot System, Task Planning
TL;DR: A novel multi-agent framework is introduced for effective collaboration among heterogeneous multi-robot systems with embodiment-aware reasoning capabilities by understanding robot URDF files, plus a novel benchmark for this task.
Abstract: Heterogeneous multi-robot systems (HMRS) have emerged as a powerful ap-
proach for tackling complex tasks that single robots cannot manage alone. Current
large-language-model-based multi-agent systems (LLM-based MAS) have shown
success in areas like software development and operating systems, but applying
these systems to robot control presents unique challenges. In particular, the ca-
pabilities of each agent in a multi-robot system are inherently tied to the physical
composition of the robots, rather than predefined roles. To address this issue,
we introduce a novel multi-agent framework designed to enable effective collab-
oration among heterogeneous robots with varying embodiments and capabilities,
along with a new benchmark named Habitat-MAS. One of our key designs is
Robot Resume: Instead of adopting human-designed role play, we propose a self-
prompted approach, where agents comprehend robot URDF files and call robot
kinematics tools to generate descriptions of their physics capabilities to guide
their behavior in task planning and action execution. The Habitat-MAS bench-
mark is designed to assess how a multi-agent framework handles tasks that require
embodiment-aware reasoning, which includes 1) manipulation, 2) perception, 3)
navigation, and 4) comprehensive multi-floor object rearrangement. The experi-
mental results indicate that the robot’s resume and the hierarchical design of our
multi-agent system are essential for the effective operation of the heterogeneous
multi-robot system within this intricate problem context.
Supplementary Material: zip
Primary Area: applications to robotics, autonomy, planning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3753
Loading