Learning Embodied Vision-Language Programming From Instruction, Exploration, and Environmental Feedback

Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Jiamu Kang, ChenCheng Jiang, Haoran Tan, Yuanhan Zhang, Kaiyang Zhou, Ziwei Liu

Primary Area: applications to robotics, autonomy, planning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Embodied AI, vision-language models

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We introduce Octopus, a novel VLM designed to proficiently decipher an agent's egocentric vision and textual task objectives and to formulate intricate action sequences and generate executable code.

Abstract: The fusion of vision and language in recent vision-language models (VLMs) represents a significant advancement in multimodal comprehension and interpretation. Furthermore, when seamlessly integrated into an embodied agent, it signifies a crucial stride towards the creation of autonomous and context-aware systems capable of formulating plans and executing commands with precision. In this paper, we introduce Octopus, a novel VLM designed to proficiently decipher an agent's egocentric vision and textual task objectives and to formulate intricate action sequences and generate executable code. Our design allows the agent to adeptly handle a wide spectrum of tasks, ranging from mundane daily chores in simulators to sophisticated interactions in complex video games. Octopus is trained by leveraging GPT-4 to generate training data, i.e., action blueprints and the corresponding executable code, within our experimental environment called OctoVerse, which provides instant feedback to refine the agent’s decision making. Through a series of experiments, we illuminate Octopus's functionality and present compelling results. By open-sourcing our model architecture, simulator, and dataset, we aspire to ignite further innovation and foster collaborative applications within the broader embodied AI community.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 374