Robustness Inside Out Testing

Deborah S. Katz, Milda Zizyte, Casidhe Hutchison, David Guttendorf, Patrick E. Lanigan, Eric M. Sample, Philip Koopman, Michael D. Wagner, Claire Le Goues

Abstract: Robustness testing is an important technique to reveal defects and vulnerabilities in software, especially software for Unmanned Autonomous Systems (UAS). We present Robustness Inside Out Testing (RIOT) as a technique directed at finding failures in autonomy systems that are able to be activated from external interfaces. The technique consists of four main steps: unit-level robustness testing, generalization, permeability analysis, and activation. Each of these steps yields a valuable deliverable in the testing process, and, when applied in succession, expands a unit-level bug to an external interface. RIOT has the following advantages over traditional robustness testing: it finds faults faster, it can find faults missed by traditional approaches, it identifies faults that can be triggered from inputs at an external interface, and it produces useful artifacts to aid in fault diagnosis and repair. In this paper, we outline each step of the RIOT process and provide an example of RIOT finding a bug on a real system that would not have been discovered using existing techniques.