Variational Constrained Reinforcement Learning with Application to Planning at Roundabout

Yuan Tian, Minghao Han, Lixian Zhang, Wulong Liu, Jun Wang, Wei Pan

Abstract: Planning at roundabout is crucial for autonomous driving in urban and rural environments. Reinforcement learning is promising not only in dealing with complicated environment but also taking safety constraints into account as a as a constrained Markov Decision Process. However, the safety constraints should be explicitly mathematically formulated while this is challenging for planning at roundabout due to unpredicted dynamic behavior of the obstacles. Therefore, to discriminate the obstacles' states as either safe or unsafe is desired which is known as situation awareness modeling. In this paper, we combine variational learning and constrained reinforcement learning to simultaneously learn a Conditional Representation Model (CRM) to encode the states into safe and unsafe distributions respectively as well as to learn the corresponding safe policy. Our approach is evaluated in using Simulation of Urban Mobility (SUMO) traffic simulator and it can generalize to various traffic flows.

Code: https://www.dropbox.com/sh/oo6zty99c6tclx1/AAA8RXynrE8K9SYpxzqBhv4Va?dl=0

Keywords: Safe reinforcement learning, Autonomous driving, obstacle avoidance