A Survey on Self-Play Methods in Reinforcement Learning

TMLR Paper4567 Authors

Abstract: Self-play, a learning paradigm where agents iteratively refine their policies by interacting with historical or concurrent versions of themselves or other evolving agents, has shown remarkable success in solving complex non-cooperative multi-agent tasks. Despite its growing prominence in multi-agent reinforcement learning (MARL), such as Go, poker, and video games, a comprehensive and structured understanding of self-play in non-cooperative games remains lacking. This survey fills this gap by offering a comprehensive roadmap to the diverse landscape of self-play methods in non-cooperative games. We begin by introducing the necessary preliminaries, including the MARL framework and basic game theory concepts. Then, it provides a unified framework and classifies existing self-play algorithms within this framework. Moreover, the paper bridges the gap between the algorithms and their practical implications by illustrating the role of self-play in different non-cooperative scenarios. Finally, the survey highlights open challenges and future research directions in self-play.

Submission Length: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Marc_Lanctot1

Submission Number: 4567