MOTSC: Model-based Offline Traffic Signal Control

Yicheng Liu, Chumeng Liang, Zherui Huang, Guanjie Zheng

Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: traffic signal control, offline reinforcement learning, transition model, movement independent transition

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: An offline traffic signal control method based on model-based offline reinforcement learning

Abstract:

Urban areas are currently suffering from more and more severe traffic congestion. One of the most straightforward ways to relieve congestion is to optimize the control of traffic lights. Varieties of reinforcement learning (RL) methods are thus born and have shown good performance in traffic signal control. However, on one hand, the performance of the RL agents may be unstable due to limited interaction data in the early stages of training, leading to even more serious traffic congestion. On the other hand, most of the data generated by the interaction are discarded after training, leading to low data utilization. Hence, it is necessary to introduce offline reinforcement learning to traffic signal control, which trains RL policies without interaction between RL policies and the environment and fully utilizes the data collected in the past.
In this paper, we propose an offline traffic signal control method based on model-based offline reinforcement learning. We formulate offline policy optimization under traffic signal control and design the transition model. A theoretical proof has been given out that our method can estimate the state of out-of-distribution samples more accurately. We conduct extensive experiments to compare our method with methods of traffic signal control and offline reinforcement learning under offline traffic signal control, where our method achieves better performance on various metrics.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2071