BasicTS: An Open Source Fair Multivariate Time Series Prediction Benchmark

Yubo Liang, Zezhi Shao, Fei Wang, Zhao Zhang, Tao Sun, Yongjun Xu

Abstract: Multivariate Time Series (MTS) is ubiquitous in the real world, and its prediction plays a vital role in a wide range of applications. Recently, many researchers have made persistent efforts to design powerful models. For example, Spatial-Temporal Graph Neural Networks (STGNNs) have become increasingly popular MTS prediction methods due to their state-of-the-art performance. However, we found there exists much unfairness in the comparison of the performance of existing models, which may prevent researchers from making correct judgments. Meanwhile, researchers usually have to build training pipelines that are complex and error-prone when designing new models, which further obstacles the quick and deep innovation in the MTS prediction field. In this paper, we first analyze the sources of unfairness and then propose a fair and easy-to-use benchmark, BasicTS, to address the above two issues. On the one hand, for a given MTS prediction model, BasicTS evaluates its ability based on rich datasets and standard pipelines. On the other hand, BasicTS provides users with flexible and extensible interfaces to facilitate convenient designing and exhaustive evaluation of new models. In addition, based on BasicTS, we provide performance revisits of several popular MTS prediction models. The proposed benchmark is publicly available at https://github.com/zezhishao/BasicTS.