Testing Finalization Mechanisms with Heap Fuzzing

Guillermo Polito

Keywords: fuzzing, automated testing, garbage collectors, finalization, ephemerons

TL;DR: We propose to fuzz heaps by generating large sequences of random heap events, to automatically find GC bugs.

Abstract:

Producing robust memory manager implementations is a challenging task. Defects in garbage collection algorithms produce subtle effects that are revealed later in program execution as memory corruptions. This problem is exacerbated by the fact that garbage collection algorithms deal with low-level aspects to be efficient. Finding, reproducing, and debugging such bugs is complex and time-consuming.
In this article, we propose to fuzz heaps by generating large sequences of random heap events. Randomly generated events exercise the garbage collection algorithm with the objective of generating VM crashes and finding bugs. Once a bug is found, we use a test case reduction algorithm that finds the smaller subset of events that reproduces the issue.
We implemented our approach on top of the VM simulator of the Pharo VM. We used it to test the ephemeron finalization mechanism in Pharo’s sequential stop-the-world generational scavenger. Our prototype found two bugs in Pharo’s ephemeron implementation. We show how such test cases were automatically reduced to trivial sequences that were easy to debug.