Controllable Neural Reconstruction for Autonomous Driving

Máté Tóth, Péter Kovács, Zoltan Bendefy, Zoltan Hortsin, Tamas Matuszka

Abstract: Neural scene reconstruction is gaining importance in autonomous driving, especially for closed-loop simulation of real-world recordings. This paper introduces an automated pipeline for training neural reconstruction models, utilizing sensor streams captured by a data collection vehicle. Subsequently, these models are deployed to replicate a virtual counterpart of the actual world. Additionally, the scene can be replayed or manipulated in a controlled manner. To achieve this, our in-house simulator is employed to augment the recreated static environment with dynamic agents, managing occlusion and lighting. The simulator’s versatility allows for various parameter adjustments, including dynamic agent behavior and weather conditions.