HomieBot: an Adaptive System for Embodied Mobile Manipulation in Open Environments

Dongping Li, Tielong Cai, Tianci Tang, Wenhao Chai, Katherine Rose Driggs-Campbell, Hongwei Wang, Gaoang Wang

Keywords: Embodied Agent, Embodied AI Benchmark, Mobile Manipulation

Abstract: Embodied Mobile Manipulation in Open Environments (EMMOE) is the challenge that agents understanding user instructions and executing long-horizon everyday tasks in home environments. This challenge encompasses task planning, decision-making, navigation and manipulation, and is crucial to develop a powerful home assistant capable of autonomously completing daily tasks. However, the absence of a holistic benchmark, data incompatibility between large language models (LLMs) and mobile manipulation tasks, the lack of a comprehensive framework, and insufficient dynamic adaptation mechanisms all continue to hinder its development. To address these issues, we propose EMMOE, the first unified benchmark that simultaneously evaluates high-level planners and low-level policies, and new metrics for more diverse evaluation. Additionally, we manually collect EMMOE-100, the first everyday task dataset featuring detailed decision-making processes, Chain-of-Thought (CoT) outputs, feedback from low-level execution and a trainable data format for Large Multimodal Models (LMMs). Furthermore, we design HomieBot, a sophisticated agent system which integrates LMM with Direct Preference Optimization (DPO) as the high-level planner, small navigation and manipulation models as the low-level executor. Finally, we demonstrate HomieBot's performance and methods for evaluating different models and policies.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8263