Mixed-Autonomy Traffic Control with Proximal Policy Optimization

Haoran Wei, Xuanzhang Liu, Lena Mashayekhy, Keith Decker

Abstract: This work studies mixed-autonomy traffic optimization at a network level with Deep Reinforcement Learning (DRL). In mixed-autonomy traffic, a mixture of connected autonomous vehicles (CAVs) and human driving vehicles is present on the roads at the same time. We hypothesize that controlling distributed CAVs at a network level can outperform the individually controlled CAVs. Our goal is to improve traffic fluidity in terms of the vehicle's average velocity and collision avoidance. We propose three distributed learning control policies for CAVs in mixed-autonomy traffic using Proximal Policy Optimization (PPO), a policy gradient DRL method. We conduct the experiments with different traffic settings and CAV penetration rates on the Flow framework, a new open-source microscopic traffic simulator. The experiments show that network-level RL policies for controlling CAVs outperform the individual-level RL policies in terms of the total rewards and the average velocity.