Diversity Is Not All You Need: Training A Robust Cooperative Agent Needs Specialist Partners
Keywords: multi-agent, reinforcement learning, robustness, diversity, specialization
TL;DR: We show that partners' specialization is crucial for training a robust generalist and propose a method that reduces overfitness of XP-min partners that already have good specialization
Abstract: Partner diversity is known to be crucial for training a robust generalist cooperative agent. In this paper, we show that partner specialization, in addition to diversity, is crucial for the robustness of a downstream generalist agent. We propose a principled method for quantifying both the diversity and specialization of a partner population based on the concept of mutual information. Then, we observe that the recently proposed cross-play minimization (XP-min) technique produces diverse and specialized partners. However, the generated partners are overfit, reducing their usefulness as training partners. To address this, we propose simple methods, based on reinforcement learning and supervised learning, for extracting the diverse and specialized behaviors of XP-min generated partners but not their overfitness. We demonstrate empirically that the proposed method effectively removes overfitness, and extracted populations produce more robust generalist agents compared to the source XP-min populations.
Primary Area: Reinforcement learning
Submission Number: 5256
Loading