EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environments

TMLR Paper5983 Authors

Abstract: Developing autonomous household robots controlled by natural language has long been a pursuit of humanity. While advancements in large language models (LLMs) and embodied intelligence make this goal closer, several challenges persist: the lack of a robotic task and benchmark that are well-aligned with realistic household tasks, limited evaluation methods and metrics, data incompatibility between LLMs and mobile manipulation trajectories. To address these issues, we propose Embodied Mobile Manipulation in Open Environments (EMMOE), a benchmark that requires agents to interpret user instructions and execute long-horizon everyday tasks in continuous space. EMMOE seamlessly integrates high-level and low-level embodied tasks into a unified framework, along with three new metrics for more diverse assessment. Additionally, we collect EMMOE-100, which features in various task attributes, detailed process annotations, re-plans after failures, and two sub-datasets for LLM training. Furthermore, we design HomieBot, a sophisticated agent system consists of LLM with Direct Preference Optimization (DPO), light weighted navigation and manipulation models, and multiple error detection mechanisms. Finally, we demonstrate HomieBot’s performance and evaluations of different models and policies.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Wilka_Torrico_Carvalho1

Submission Number: 5983