A Case Study for the Behaviors of Generalists and Specialists in Competitive Games

Stefan Juang, Qiyang Cao, Yuan Zhou, Ruochen Liu, Elvis S. Liu, Nevin L. Zhang

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Population Learning, Information Theory, Reinforcement Learning, Competitive Learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: In this study, we investigate the behavioral change of a heterogeneous population as a system of information exchange. Previous approaches, such as OpenAIFive and NeuPL, have modeled a population as a single conditioned neural network to achieve rapid competitive learning. However, we found that this approach can overgeneralize the population as Generalists and hinder individual learning of specializations. To address this challenge, we propose Joint Entropy Minimization (JEM), a novel policy gradient formulation for heterogeneous populations. Our theoretical and experimental results show that JEM enables the training of Generalist populations to become Specialists. Compared to previous methods, Specialists trained with JEM exhibit increased strategy diversity, improved competitive performance, and reduced population performance disparity. These findings suggest that modeling a heterogeneous population as a group of Specialists can more fully realize the diverse potential of individual agents.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4825