A Strategy for Boundary Adherence and Exploration in Black-Box Testing of Autonomous Vehicles

John M. Thompson, Quentin Goss, Mustafa Ilhan Akbas

Abstract: The validation of artificial intelligence (AI) controlled vehicles is a vexing challenge. Black box testing of decision making in these vehicles has been used to abstract out the inner complexity of the AI. In scenario-based black box testing, the AI is placed within a scenario, and the input space for that scenario is explored. The fundamental metric of “how well tested the system is for that scenario” is based on the input state space coverage. Since most of these spaces have a high number of dimensions, it is critical to sample the state space efficiently and identify performance boundaries for the vehicle under test. In this paper, we propose a boundary adherence approach for autonomous vehicle validation that can explore the boundary between targeted and non-targeted behavior. This paper significantly improves and extends our previous approach that focused on generic black-box testing of AI systems by optimizing the algorithm itself, adding new tools for exploration, and applying the strategy to scenario-based AV testing. We provide an example regression of a scenario which illustrates the ability to model boundaries after they have been explored. Further results on higher dimensions show differing adherence strategies can improve exploration efficiency and how boundary exploration focuses on more “interesting” scenarios. Upon exploring the boundary, we found that predictions can be made about whether or not the system will result in targeted behavior.