Unsupervised Online Anomaly Detection With Parameter Adaptation for KPI Abrupt Changes

Guang Yu, Zhiping Cai, Siqi Wang, Haiwen Chen, Fang Liu, Anfeng Liu

Abstract: IT companies need to monitor various Key Performance Indicators (KPIs) and detect anomalies in real time to ensure the quality and reliability of Internet-based services. However, due to the diversity of KPIs, the ambiguity and scarcity of anomalies and the lack of labels, anomaly detection for various KPIs has been a great challenge. Existing KPI anomaly detection methods have not explored the properties of anomalies in KPIs in detail to our best knowledge. Therefore, we explore anomalies in KPIs and recognize a common and important form of anomalies named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">abrupt changes</i> , which often indicate potential failures in the relevant services. For <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">abrupt changes</i> in various KPIs, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDCOL</i> , an unsupervised online anomaly detection algorithm with parameter adaptation from the perspective of anomalies for the first time. We propose three techniques: high order <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${D}$ </tex-math></inline-formula> ifference extraction and combination, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${D}$ </tex-math></inline-formula> ensity-based <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${C}$ </tex-math></inline-formula> lustering with parameter adaptation and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${O}\text{n}{L}$ </tex-math></inline-formula> ine detection with subsampling ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDCOL</i> ). Compared with traditional statistical methods and unsupervised learning methods, extensive experimental results and analysis on a large number of public KPIs show the competitive performance of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDCOL</i> and the significance of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">abrupt changes</i> . Furthermore, we provide an interpretation for the promising results, which shows that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDCOL</i> can be robust to KPI expected concept drifts, and obtain a good feature distribution of normal data in KPIs.