Pierce: A Testing Tool for Neural Network Verification Solvers

Joseph Scott, Guanting Pan, Piyush Jha, Elias B. Khalil, Vijay Ganesh

Abstract: We introduce Pierce, a versatile and extensible testing tool aimed at solvers for the neural network verification (NNV) problem. At its core, Pierce implements a fuzzing engine over the Open Neural Network Exchange (ONNX) – a standardized model format for deep learning and classical machine learning, and VNN-LIB – a specification standard over the input-output behavior of machine learning systems. Pierce supports the entirety of the VNN-LIB and most of ONNX v18. The API of Pierce is designed to enable users to create a variety of software testing tools, such as performance and mutation fuzzers, as well as delta debuggers, with relative ease. For example, Pierce provides a rich generator for computation graphs and specifications that allows users to easily specify a wide variety of configurations, as well as mutators that ensure that mutated computation graphs are well-formed.