Multimodal Reinforcement Learning with Dynamic Graph Representations for Autonomous Driving Decision-Making

Ting Su, Chun-Yang Zhang

Abstract: It is crucial for autonomous vehicles to make safe and effective decisions in real-time dynamic road environments through decision-making systems. Traditional rule-based decision-making methods struggle to handle complex and variable traffic conditions, limiting their reliability. Data-driven methods such as imitation learning (IL) and reinforcement learning (RL) offer better generalization and adaptability. However, existing methods often fail to capture comprehensive and accurate representations of traffic scenes, especially in high-traffic areas like roundabouts, where neglecting the impact of other traffic participants poses significant safety risks. To address this issue, we propose a multimodal reinforcement learning framework based on integrated dynamic graph representation learning (DGMRL). Our framework designs a spatial-temporal-coupled dynamic graph neural network, which implicitly models temporal information and integrates interaction information and temporal evolution in a unified manner. Additionally, we employ a cross-attention fusion mechanism to effectively integrate multimodal data, constructing a comprehensive driving environment representation. Extensive experimental validation demonstrates that our method outperforms existing baseline models across diverse driving scenarios, with particularly significant performance improvements in traffic-dense environments.

Submission Number: 174