MineDojo: Building Open-Ended Embodied Agents with Internet-Scale KnowledgeDownload PDF

Published: 17 Sept 2022, Last Modified: 03 Jul 2024NeurIPS 2022 Datasets and Benchmarks Readers: Everyone
Keywords: Embodied Agents, Minecraft, Open-ended Learning, Multitask Learning, Internet Knowledge Base, Reinforcement Learning, Large Pre-training
TL;DR: MineDojo is a new framework built on the Minecraft game for developing open-ended, generally capable embodied agents.
Abstract: Autonomous agents have made great strides in specialist domains like Atari games and Go. However, they typically learn tabula rasa in isolated environments with limited and manually conceived objectives, thus failing to generalize across a wide spectrum of tasks and capabilities. Inspired by how humans continually learn and adapt in the open world, we advocate a trinity of ingredients for building generalist agents: 1) an environment that supports a multitude of tasks and goals, 2) a large-scale database of multimodal knowledge, and 3) a flexible and scalable agent architecture. We introduce MineDojo, a new framework built on the popular Minecraft game that features a simulation suite with thousands of diverse open-ended tasks and an internet-scale knowledge base with Minecraft videos, tutorials, wiki pages, and forum discussions. Using MineDojo's data, we propose a novel agent learning algorithm that leverages large pre-trained video-language models as a learned reward function. Our agent is able to solve a variety of open-ended tasks specified in free-form language without any manually designed dense shaping reward. We open-source the simulation suite, knowledge bases, algorithm implementation, and pretrained models (https://minedojo.org) to promote research towards the goal of generally capable embodied agents.
URL: https://minedojo.org
Dataset Url: Please find the dataset access instructions and guidelines on https://minedojo.org and our supplementary material.
Author Statement: Yes
License: Simulator and benchmarking suite code: MIT license. MineDojo Youtube database: Creative Commons Attribution 4.0 International (CC BY 4.0). MineDojo Wiki database: Creative Commons Attribution Non Commercial Share Alike 3.0 Unported. MineDojo Reddit database: Creative Commons Attribution 4.0 International (CC BY 4.0).
Supplementary Material: zip
Contribution Process Agreement: Yes
In Person Attendance: Yes
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 5 code implementations](https://www.catalyzex.com/paper/minedojo-building-open-ended-embodied-agents/code)
24 Replies