TeamCraft: A Benchmark for Embodied Multi-Agent Systems in Minecraft

Qian Long, Zhi Li, Ran Gong, Ying Nian Wu, Demetri Terzopoulos, Xiaofeng Gao

Keywords: Multi-agent system, embodied AI

TL;DR: A multi-modal Multi-agent benchmark based on Minecraft.

Abstract: Complex 3D environments replete with dynamic interactions among multiple agents and objects are essential for the development of embodied intelligent agents. To facilitate research on Multi-Agent (MA) systems, we introduce \benchmark, a challenging MA benchmark based on the Minecraft game. Instead of the abstract vector inputs commonly provided to agents in MA systems research, \benchmark provides agents with multi-modal task specifications and observations. Given the three-orthographic-view graph of the environment along with language instructions, the agents must efficiently collaborate to complete assigned tasks. Such multi-modal inputs pose a higher level of difficulty, since agents must generalize across diverse object and background imagery, different numbers of agents, a wide range of tasks, etc. Our planner-generated dataset includes various tasks, such as building construction, smelting, and farming, with a total of 70,000 procedurally-generated demonstrations that feature over 50 objects across a wide variety of scenes. We test the generalization abilities of several baseline Vision-Language Model (VLM) multi-agent control strategies in centralized and decentralized settings.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8575