AR-CP: Uncertainty-Aware Perception in Adverse Conditions with Conformal Prediction and Augmented Reality For Assisted Driving

Achref Doula, Max Mühlhäuser, Alejandro Sánchez Guinea

Abstract: Deep learning models are pivotal in enhancing driver assistance systems and improving environmental perception. However, the tendency of neural networks towards overconfident predictions poses a risk of inaccurate predictions, potentially compromising driver safety in adverse conditions. To mitigate this issue, we introduce AR-CP, an uncertainty-aware framework designed to augment driver perception in scenarios characterized by adverse weather and insufficient lighting, through the integration of conformal prediction and augmented reality (AR). Our framework initiates with a conformal prediction step that produces an uncertainty-aware prediction set including potential object classes at a predefined probability level. Subsequently, AR is used to provide a simplified and informative visualization of the closest common parent class of the classes in the prediction set, thereby reducing the likelihood of misinformation. We provide a principled formulation and theoretical analysis of our framework. We evaluate AR-CP on the ROAD dataset, a large dataset containing different difficult situations that induce high uncertainty during prediction time. The results show that our framework outperforms state-of-the-art approaches in providing smaller prediction sets while holding the theoretical guarantees, ensuring an uncertainty-aware prediction, and reducing user confusion. We conduct an immersive user study with 15 participants to investigate the effects of our concept on the quality of perception, situation awareness, and mental load of participants. The results show that our concept facilitates a safer driving experience while holding the mental load low and the situation awareness high.