COMRAD: A Benchmark for Embodied Cooperative Multi-Agent Reinforcement Learning

Khoi H.B. Nguyen, Dimitar Zhivkov Zhekov, Tristan Tomilin

Keywords: Cooperation, Reinforcement Learning, MARL, Benchmark, Egocentric, 3D, Doom, Vision, Perception, Embodied AI

TL;DR: A cooperative multi-agent reinforcement learning benchmark in complex, egocentric environments.

Abstract: Benchmarks are central to the development and evaluation of multi-agent reinforcement learning (MARL) algorithms. As the cooperative MARL community has grown, two categories of evaluation environments have proven indispensable: low-dimensional feature-vector benchmarks that isolate algorithmic behavior in compact state spaces, and two-dimensional pixel-based benchmarks that rely on overhead visual observations. However, to bridge the gap to real-world embodied settings such as robotics and autonomous navigation, algorithms must handle high-dimensional, 3D egocentric visual complexity. While various 3D environments have been explored for vision-based RL, no existing platform simultaneously provides a standardized, purely cooperative multi-agent benchmark with 3D first-person observations and high-throughput simulation. To address this gap, we introduce $\textbf{COMRAD}$, a $\textbf{CO}$operative $\textbf{M}$ulti-Agent $\textbf{R}$einforcement Le$\textbf{A}$rning benchmark suite in $\textbf{D}$oom, featuring a diverse set of challenging scenarios spanning role asymmetry, temporal synchronization, and spatial navigation. To introduce within-scenario variability, we develop $\textbf{DoomGen}$, a procedural map generator that produces diverse layout configurations for every scenario. We integrate COMRAD with Sample Factory, a high-throughput asynchronous RL framework, and implement seven MARL baselines on top of it, reaching ${\sim}25\mathrm{K}$ frames per second during training. Our experiments show that COMRAD poses significant challenges for current CTDE methods, establishing visual cooperative MARL as an important open frontier.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Paper Type: Standard paper

Submission Number: 73