Exposing Complex Bug-Triggering Conditions in Distributed Systems via Graph Mining

Eunsoo Seo, Mohammad Maifi Hasan Khan, Prasant Mohapatra, Jiawei Han, Tarek F. Abdelzaher

Abstract: Software bugs in distributed systems are notoriously hard to find due to the large number of components involved and the non-determinism introduced by race conditions between messages. This paper introduces Pop Mine, a tool for diagnosing corner-case bugs by finding the minimal causal directed acyclic graph (DAG) of events, spanning multiple processes, which captures a bug-triggering condition. Being based on causal order, a global notion of time is not required in uncovering bug-triggering distributed event patterns. Bug triggering event DAGs can be identified by comparing execution graphs from successful runs to those where bug manifestations were observed, and exposing the minimal discriminative event DAGs that may be responsible for the problem. This is a significant extension to prior debugging tools, in that prior work considered much simpler bug-triggering conditions such as single events, event sets, or ordered chains of events. To the authors' knowledge, this is the first paper that considers bug-triggering conditions in the form of distributed event graphs. To prove the effectiveness of our approach, we applied our tool to VCP, Chord and GreenGPS and diagnosed bugs. We also present performance analysis results to demonstrate the scalability of our approach.