UMAP: A Highly Extensible and Physics-Based Simulation Environment for Multi-agent Reinforcement Learning

Tianyi Hu, Qingxu Fu, Zhiqiang Pu, Yuan Wang, Tenghai Qiu

Keywords: multi-agent reinforcement learning, simulation environment, reinforcement learning

Abstract:

Existing simulation environments in the field of multi-agent reinforcement learning (MARL) either lack authenticity or complexity. The data generated by these environments significantly deviate from the requirements of the real world, hindering the practical application of MARL. To address this issue, we propose Unreal Multi-Agent Playground (UMAP), a highly extensible, physics-based 3D simulation environment implemented on the Unreal Engine. UMAP is user-friendly in terms of deployment, modification, and visualization, and all its components are open-sourced. Based on UMAP, we design a series of MARL tasks featuring heterogeneous agents, large-scale agents, multiple teams, and sparse team rewards.
We also develop an experimental framework compatible with algorithms ranging from
rule-based to MARL-based provided by third-party frameworks. In the experimental section, we utilize the designed tasks to test several state-of-the-art algorithms. Additionally, We also conduct a physical experiment to demonstrate UMAP's potential in sim-to-real applications, which is a significant advantage due to the high extensibility and authenticity of UMAP. We believe UMAP can play an important role in the MARL field by evaluating existing algorithms and helping them apply to real-world scenarios, thus advancing the field of MARL.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6491