Ghost in the Minecraft: Hierarchical Agents for Minecraft via Large Language Models with Text-based Knowledge and Memory

Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, Jifeng Dai

Primary Area: applications to robotics, autonomy, planning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Minecraft, Large Language Models, Game Agents, In-context Learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: As modern computer games continue to evolve, there is a growing need for adaptive agents that can effectively navigate, make decisions, and interact within vast, ever-changing worlds. While recently developed agents based on Large Language Models (LLMs) show promise in adaptability for controlled text environments, expansive and dynamic open worlds like Minecraft still pose challenges for their performance. To address this, we introduce Ghost in the Minecraft (GITM), a novel hierarchical agent that integrates LLMs with text-based knowledge and memory. Structured actions are constructed to enable LLMs to interact in Minecraft using textual descriptions, bridging the gap between desired agent behaviors and LLM limitations. The hierarchical agent then decomposes goals into sub-goals, actions, and operations by leveraging text knowledge and memory. A text-based in-context learning method is also designed to enhance future planning. GITM demonstrates the potential of LLMs in Minecraft's evolving open world. Notable milestones are collecting 99.2\% of items and a 55\% success rate on the popular ``ObtainDiamond'' task. GITM also shows impressive learning efficiency, requiring minimal computational resources.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7382