Cost-aware Discovery of Contextual Failures using Bayesian Active Learning

Anjali Parashar, Joseph Zhang, Yingke Li, Chuchu Fan

Keywords: Failure discovery, Testing, Contextual failures

TL;DR: We propose a method for contextual failure discovery of robotic systems with high cost of evaluation using a Bayesian Active Learning approach

Abstract: Ensuring the robustness of robotic systems is crucial for their deployment in safety-critical domains. Failure discovery, or falsification, is a widely used approach for evaluating robustness, with recent advancements focusing on improving sample efficiency and generalization through probabilistic sampling techniques and learning-theoretic approaches. However, existing methods typically rely on explicitly defined analytical cost functions to characterize failures, often overlooking the underlying causes and diversity of discovered failure scenarios. In this work, we propose a novel failure discovery framework that integrates contextual reasoning in the falsification process, specifically tailored for high evaluation-cost applications. Our method incorporates expert-in-the-loop feedback to construct a probabilistic surrogate model of failures using Bayesian inference. This model is iteratively refined and leveraged to guide an active learning strategy that prioritizes the discovery of diverse failure cases. We empirically validate our approach across a range of tasks for high-cost contextual falsification in robotic manipulation and autonomous driving.

Submission Number: 670