Memory Enhanced Context Awareness for Large Language Model Based Autonomous Driving

Yanshuo Zheng, Hanwen Zhang, Dagang Li

Abstract: In this paper, we utilize GLM-4, a large language model, for decision-making in the realm of autonomous driving and compare its performance with the Deep Q-Network (DQN) and Graph Representation for Autonomous Driving (GRAD) method from reinforcement learning. Autonomous driving is a highly complex task, it needs the ability to make decisions in diverse environments. Thus, we aim to assess the potential of GLM-4 in various situations. GLM-4 has vast pre-trained knowledge that can have an inference based on contextual information. Our approach integrates the GLM-4 with the memory module that stores past experiences and leverages the pre-trained knowledge of GLM-4 to enhance decision-making under different driving scenarios. We tested the memory module with 5, 20, and 40 memory items, and conducted experiments using 1-shot and 3-shots experiences to analyze how accumulated experience influences the model's performance.