Reinforcement Compiler Fuzzing

Xiao Liu, Rupesh Prajapati, Xiaoting Li, Dinghao Wu

Keywords: Compiler Testing, Fuzz Testing, Deep Q-learning, Coverage-guided Testing

Abstract: To enforce the correctness of compilers is important for every computing system. Fuzzing is an efficient way to find security vulnerabilities by repeatedly testing programs with randomly modified input data. However, in the context of compilers, fuzzing is challenging because the inputs are pieces of codes which are supposed to be both syntactically and semantically valid to pass front-end checks. Moreover, the fuzzed inputs should be distinct to trigger abnormal crashes, memory leaks or failing assertions that not being detected before. In this paper, we proposed an automatic code synthesis framework called FuzzBoost based on reinforcement learning. By adopting testing coverage information collected from runtime traces as the reward, we propose a learning system with the state-of-the-art deep Q-learning algorithm that optimizes this reward. In this way, the fuzzing agent learns the actions to perform to fuzz a seed program that achieves an overall goal of testing coverage improvement. We have implemented this new approach and preliminary evidence shows that reinforcement fuzzing can outperform baseline random fuzzing on production compilers (i.e. GCC).