Shrinking Counterexamples in Property-Based Testing with Genetic Algorithms

Fang-Yi Lo, Chao-Hong Chen, Ying-Ping Chen

Abstract: In this paper, genetic algorithms are proposed to shrink counterexamples found by QuickChick, a property-based testing framework for Coq. In order to make the outcome of property-based testing humanly understandable and inspectable, genetic algorithms are brought into the realm of rigorous software development as shrinkers capable of handling a broad range of data structures. In the present study, two showcases, merge sort and insertion of red-black trees, are investigated for illustrative purposes. Due to the lack of relevant results existing in the literature, two baseline methods, random sample and random walk are included in the experiments for comparison with the proposed genetic algorithm. The obtained results indicate that the proposal is effective since the program mistake can be identified with ease by examining the shrunk counterexamples and also that the adopted genetic algorithm statistically significantly outperforms random sample and random walk in both counterexample sizes and running time.