Action Prediction for Cooperative Exploration in Multi-agent Reinforcement Learning

Yanqiang Zhang, Dawei Feng, Bo Ding

Abstract: Multi-agent reinforcement learning methods have shown significant progress, however, they continue to exhibit exploration problems in complex and challenging environments. To address the above issue, current research has introduced several exploration-enhanced methods for multi-agent reinforcement learning, they are still faced with the issues of inefficient exploration and low performance in challenging tasks that necessitate complex cooperation among agents. This paper proposes the prediction-action Qmix (PQmix) method, an action prediction-based multi-agent intrinsic reward construction approach. The PQmix method employs the joint local observation of agents and the next joint local observation after executing actions to predict the real joint action of agents. The method calculates the action prediction error as the intrinsic reward to measure the novel of the joint state and encourages agents to actively explore the action and state spaces in the environment. We compare PQmix with strong baselines on the MARL benchmark to validate it. The result of experiments demonstrates that PQmix outperforms the state-of-the-art algorithms on the StarCraft Multi-Agent Challenge (SMAC). In the end, the stability of the method is verified by experiments.