MACS: Multi-Agent Adversarial Reinforcement Learning for Finding Diverse Critical Driving Scenarios

Shuting Kang, Qian Dong, Yunzhi Xue, Yanjun Wu

Abstract: Critical scenario generation plays a crucial role in the autonomous driving test by efficiently and effectively identifying various hazardous scenarios to evaluate the multiagent system under test. The performance of existing solution models is hampered by sparse rewards resulting from long time-steps in driving scenarios. Moreover, they fail to guide the generation of more diverse scenarios because of the lack of a fine-grained design. To efficiently and effectively discover various critical scenarios, we propose the MACS method based on multiagent reinforcement learning to guide adversaries foiled the agent under test by replay buffer optimization and objective function design. By adopting the hindsight experience replay method, historical experiences are reused to address the challenge of sparse rewards and improve sample efficiency. Furthermore, we integrate the entropy term into the objective function to explore different driving strategies, thereby leading to the creation of diverse scenarios. We have achieved a new state-of-the-art performance in evaluating rule-based agents using an industrial-grade platform, SMARTS. The experimental results demonstrate that MACS can effectively generate diverse critical scenarios that lead to the failure of the agent under test. We also apply cluster methods, including DBSCAN and TRACLUS, to conduct diversity analysis of the generated scenarios. Besides, we evaluate and improve the reinforcement learning decision algorithm for the vehicle under test with our generated scenarios and give empirical conclusions about its robustness.