Higher-order Markov Graph based Bug Detection in Cloud-based Deployments

Qing Cao, Haoran Niu

Abstract: Detecting execution anomalies is an integral part of building and protecting modern large-scale distributed systems. These systems generate a large volume of system logs to record system state and significant events, which provide a valuable resource to help debug system failures and perform root cause analysis. However, detecting anomalies in log sequences remains a challenge due to reasons including the imbalance of the data, the complexity of relationships between events, and the high dimensionality of log events. Traditional graph-based models may lose important higher-order sequence patterns and result in undetectable higher-order anomalies because they use first-order or fixed-order networks to represent the underlying log data. In this paper, we propose a novel unsupervised graph-based anomaly detection method, called GraphLog, which utilizes a variable high-order network representation. This variable representation enables GraphLog to efficiently learn log patterns from normal logs and detect first-order and higher-order log patterns that deviate from normal data. We demonstrate that the proposed graph-based log anomaly detection algorithm is effective, and it outperforms other baseline methods when trained using two real-world datasets.