Autonomous Driving Through Deep Learning in Video Games: A Visual-Based Perception and Action Approach

Anestis Kastellos, Socratis Gkelios, Savvas A. Chatzichristofis, Pericles Leng Cheng

Abstract: Recent studies have demonstrated the utility of simulations and video games as viable approximations for real-world problems, particularly in autonomous driving. Initial testing and validation of autonomous systems are typically conducted in simulated environments, as they effectively mimic human behavior's visual-based perception and action. This paper presents a deep-learning model for autonomous driving in a video game that can close the loop between sensing and actuation. Our algorithm utilizes a pre-trained deep neural network to process images of the game environment and output the optimal driving commands. We use only visual information from the game, without additional processed data, and present an extensive comparison of model variations. Our approach makes a valuable contribution to the field of autonomy by showing that it can achieve responsive and realistic driving behavior using only visual input. By leveraging the power of video games and deep learning, autonomous driving can be significantly improved, making transportation safer and more reliable.