Uncertainty-Calibrated Closed-Loop Simulation for Autonomous Driving Evaluation

Mahule Roy, Subhas Roy

Keywords: autonomous driving, simulation, uncertainty calibration, risk-aware evaluation, reinforcement learning, nuScenes, trajectory prediction, LiDAR modeling

TL;DR: We propose an uncertainty-calibrated autonomous driving simulator that models agent and sensor uncertainty, introduces a risk-weighted evaluation metric, and improves policy robustness in rare scenarios.

Abstract: Simulation is a cornerstone of autonomous vehicle (AV) development, yet most existing simulators produce either deterministic outputs or stochastic outputs that are not properly calibrated. This paper addresses the critical issue of unreliable confidence estimation in simulation by proposing an uncertainty-calibrated framework. Our approach explicitly models and propagates epistemic uncertainty in both agent behavior and sensor simulation. We introduce a risk-aware closed-loop evaluation protocol, along with a novel metric—Risk-Weighted Simulation Error (RWSE)—which incorporates the simulator’s confidence into performance assessment. Experiments using scenarios derived from the nuScenes dataset demonstrate that our calibrated simulator provides more reliable safety assessments: collision risk estimates in rare scenarios reduce underestimation bias from 37% to 5% compared to proxy real-world risk indicators. Additionally, reinforcement learning policies trained in this framework achieve improved robustness, with a 14.8% relative increase in success rate and a 44% reduction in catastrophic failures (p < 0.05). Our analysis shows that deterministic simulators, due to their lack of behavioral stochasticity, tend to produce artificially low collision rates and fail to capture rare, high-risk events. Although our ensemble-based approach incurs a computational overhead (approximately 3.2× slower), the resulting gains in evaluation reliability justify this cost. Overall, these results highlight that uncertainty calibration is a crucial complement to realism for trustworthy simulation-based evaluation and training of autonomous vehicles.

Submission Number: 1