VistaRAG: Toward Safe and Trustworthy Autonomous Driving Through Retrieval-Augmented Generation

Xingyuan Dai, Chao Guo, Yun Tang, Haichuan Li, Yutong Wang, Jun Huang, Yonglin Tian, Xin Xia, Yisheng Lv, Fei-Yue Wang

Abstract: Autonomous driving based on foundation models has recently garnered widespread attention. However, the risk of hallucinations inherent in foundation models could compromise the safety and reliability of autonomous driving systems. This letter, as part of a series of reports from the Distributed/Decentralized Hybrid Workshop on Foundation/Infrastructure Intelligence (DHW-FII), aims to tackle these issues. We introduce VistaRAG, which integrates retrieval-augmented generation (RAG) technologies into autonomous driving systems based on foundation models, to address the inherent reliability challenges in decision-making. VistaRAG employs a dynamic retrieval mechanism to access highly relevant driving experience, real-time road network status, and other contextual information from external databases. This aids foundation models in informed reasoning and decision-making, thereby enhancing the safety and trustworthiness of foundation-model-based autonomous driving systems under complex traffic scenarios.