Reproducibility Study of “Efficient Episodic Memory Utiliza- tion of Cooperative Multi-Agent Reinforcement Learning"

TMLR Paper4314 Authors

Abstract: This paper reports on the reproducibility study on the paper "Efficient episodic memory utilization of cooperative multi-agent reinforcement learning" by \cite{na2024}. The original study proposed a method to enhance MARL performance by leveraging episodic memory to accelerate learning and prevent local optima convergence. EMU introduced a trainable encoder/decoder structure for memory retrieval and an episodic incentive reward mechanism to promote desirable transitions. The original work evaluated the method in StarCraft II and Google Research Football, demonstrating improvements over state-of-the-art approaches. This study further examines the effectiveness of EMU by assessing its reported performance improvements, the impact of its state embedding approach on exploration efficiency, and the robustness of its incentive mechanism in preventing suboptimal convergence. The analysis focuses on the SMAC benchmark, particularly in complex scenarios where EMU showed the most promise, while also exploring its scalability in high-performance computing environments to determine its computational feasibility. The findings confirm the advantages of EMU but underscore the sensitivity of its performance to embedding quality and hyperparameter selection. Our extended implementation and results are available on https://anonymous.4open.science/r/MLRC-EMU-E0EF/README.md.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Sebastian_Tschiatschek1

Submission Number: 4314