Benchmarking Multivariate Time Series Anomaly Detection with Large-Scale Real-World Datasets

Chaoli Zhang, Yingying ZHANG, Qingsong Wen, Lanshu Peng, Yiyuan Yang, Chongjiong Fan, Minqi Jiang, Lunting Fan, Liang Sun

Keywords: time series anomaly detection, real_world datasets, metric discussion, benchmarking

Abstract: Time series anomaly detection is of significant importance in many real-world applications, including finance, healthcare, network security, industrial equipment, complex computing systems, space probe, etc. Most of them involve multi-sensor systems, thus how to perform multivariate time series anomaly detection (MTSAD) has attracted widespread attention. This broad attention has fueled extensive research endeavors aiming to innovate and develop methods and techniques to improve the efficiency and precision of anomaly detection on multivariate time series data, including classic machine learning methods and deep learning methods. However, how to evaluate the performance of all these methods is a challenging task. The first challenge lies in the limited public benchmark datasets for MTSAD, and all of these datasets are criticized from some perspectives. The second but related challenge is, the best metric for time series anomaly detection remains unclear, which makes the research in MTSAD hard to follow. In this paper, we advance the benchmarking of multivariate time series anomaly detection from datasets, evaluation metrics, and algorithm comparison. To the best of our knowledge, we have generated the largest real-world datasets for MTSAD from the Artificial Intelligence for IT Operations (AIOps) system for a real-time data warehouse in a leading cloud computing company. We review and compare popular evaluation metrics including recently proposed ones. To evaluate classic machine learning and recent deep learning methods fairly, we have performed extensive comparisons of these methods on various datasets. We believe our benchmarking and datasets can promote reproducible results and accelerate the progress of MTSAD research.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4791