Improving Generalization of Reinforcement Learning for Multi-agent Combating Games

Kejia Wan, Xinhai Xu, Yuan Li

Abstract: Multi-agent combating games have attracted great interest from the research community of reinforcement learning. Most work concentrates on improving the performance of agents, and ignores the generality of the trained model on different tasks. This paper aims to enhance the generality of reinforcement learning, which makes the trained model easily adapt to different combating tasks with a variable number of agents. We divide the observation of an agent to the common part which is related to the number of other agents, and the individual part which includes only its own information. The common observation is designed with a special representation of matrices irrelevant to the number of agents, and we use convolutional networks to extract valuable features from multiple matrices. The extracted features and the individual observation form new input for reinforcement learning algorithms. Meanwhile, the number of agents also changes during the combating process. We introduce a death mask technique to avoid the effects of the dead agents on the loss computation of multi-agent reinforcement learning. Finally, we conducted a lot of experiments in StarCraft-II on unit micromanagement missions. It turned out that the proposed method could significantly improve the generality and transferring ability of the model between different tasks.