NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Driving

William Ljungbergh, Adam Tonderski, Joakim Johnander, Holger Caesar, Kalle Åström, Michael Felsberg, Christoffer Petersson

Abstract:

We present a versatile NeRF-based simulator for testing autonomous
driving (AD) software systems, designed with a focus on sensorrealistic
closed-loop evaluation and the creation of safety-critical scenarios.
The simulator learns from sequences of real-world driving sensor
data and enables reconfigurations and renderings of new, unseen scenarios.
In this work, we use our simulator to test the responses of AD
models to safety-critical scenarios inspired by the European New Car Assessment
Programme (Euro NCAP). Our evaluation reveals that, while
state-of-the-art end-to-end planners excel in nominal driving scenarios
in an open-loop setting, they exhibit critical flaws when navigating our
safety-critical scenarios in a closed-loop setting. This highlights the need
for advancements in the safety and real-world usability of end-to-end
planners. By publicly releasing our simulator and scenarios as an easyto-
run evaluation suite, we invite the research community to explore, refine,
and validate their AD models in controlled, yet highly configurable
and challenging sensor-realistic environments.