EgoPlan: Towards Effective Embodied Agents via Egocentric Planning

Zhirui Fang, Ming Yang, Weishuai Zeng, Junpeng Yue, Boyu Li, Jiafei Lyu, Xiu Li, Zongqing Lu

Keywords: diffusion world model, vision-language-planning, embodied decision-making agent

TL;DR: We explore leveraging large multi-modal models (LMMs) and Text2image models to build a more general embodied agent.

Abstract: We explore leveraging large multi-modal models (LMMs) and Text2image models to build a more general embodied agent. LMMs excel in planning long-horizon tasks over symbolic abstractions but struggle with grounding in the physical world, often failing to accurately identify object positions in images. A bridge is needed to connect LMMs to the physical world. The paper proposes a novel approach, egocentric vision language planning (EgoPlan), to handle long-horizon tasks from an egocentric perspective in varying household scenarios. This pipeline leverages a diffusion model to simulate the fundamental dynamics between states and actions, discusses how to integrate computer vision related techniques like style transfer and optical flow to enhance ability of modeling spatial states and generalization across different environmental dynamics. The LMM serves as a planner, breaking down instructions into sub-goals and selecting actions based on their alignment with these sub-goals, thus enabling more generalized and effective decision-making. By using LMM, we can output text actions, using a series of mechanisms such as reflection to perform high-level task decomposition and low-level action output end-to-end. Experiments show that EgoPlan improves long-horizon task success rates from the egocentric view compared to baselines across household scenarios.

Primary Area: applications to robotics, autonomy, planning

Submission Number: 12292