MARL for Traffic Signal Control in Scenarios with Different Intersection Importance

Liguang Luan, Yu Tian, Wanqing Fang, Chengwei Zhang, Wanli Xue, Rong Chen, Chen Sang

Abstract: Recent efforts that applied Multi-Agent Reinforcement Learning (MARL) to the adaptive traffic signal control (ATSC) problem have shown remarkable progress. However, those methods assume that all agents in the cooperative games are isomorphic, which ignores the situation that different agents can play heterogeneous roles in the ATSC scenario. The tolerance of vehicles at different intersections in the same area is different, e.g., traffic congestion near hospitals or schools will affect the timely treatment of patients or the safety of children and definitely need to be paid more attention than ordinary congestions. Motivated by the human wisdom in cooperative behaviours (e.g. team members will execute the action according to the strategy implemented by the team leader), we present a leader-follower paradigm based Markov game model which taking into account both the overall and special intersections. Specifically, the leader-follower paradigm control intersections in a traffic scenario by two kinds of agents, i.e., leader agent controlling intersections that need special attention, and follower agents controlling ordinary intersections. Then a multi-agent reinforcement learning framework, named Breadth First Sort Hysteretic DQN (BFS-HDQN) is proposed to train the optimal control policy of the proposed ATSC model. BFS-HDQN consists of two parts, an independent MARL algorithm (here we use Hysteretic DQN as the base algorithm) to train different kinds of agents, and a communication mechanism based on Breadth First Sort (BFS) to generate observation information of each agent. We evaluate our methods empirically in two synthetic and one real-world traffic scenarios. Experimental results show that, compared with the state-of-the-art methods, BFS-HDQN can not only ensure the optimal overall performance, but also obtain better performance at special intersections, in almost all metrics commonly used in ATSC.