A Fully Unsupervised and Efficient Anomaly Detection Approach with Drift Detection Capability

Chang How Tan, Vincent C. S. Lee, Mahsa Salehi, Slaven Marusic, Srimal Jayawardena, Dion Lucke

Abstract: Accurate detection of anomalies on dynamic data streams is challenging due to the high velocity, large volume, and particularly, its dynamicity property which exhibits concept drift. This may result in varying anomaly contexts over time. Some researches assume such dynamicity is seasonal and attempt to model the seasonality for anomaly detection. However, this becomes non-viable when the assumption is violated. In this paper, we propose MIRMAD, a simple, on-line, and ensemble-based anomaly detection algorithm that is able to overcome the above-mentioned challenges with the capability to identify drifting locations in the data stream and discard outdated data to minimize performance losses. Especially in an unsupervised environment, identifying the drift locations provides an additional level of information to analysts for informed decision-making. Empirical results on benchmark data sets have demonstrated that the fully unsupervised MIRMAD’s performances are comparable to even semi-supervised approaches and yet runs at least 15 times faster than the compared methods on average. We further investigate MIRMAD’s efficacy in a real-world case study and provided a detailed sensitivity analysis on different parameter settings.