MuSED-FM: A Benchmark for Evaluating Multivariate Time Series Foundation Models

Caleb Chuck, Sai Shankar Narasimhan, Shubhankar Agarwal, Aditya Narayanan, Avantika Gupta, Raghav Mallampalli, Fan Feng, Anthony Bao, Jeffrey B. Lai, William Gilpin, Sandeep P. Chinchali, sujay sanghavi

Keywords: Multivariate Tme Series, Foundation Models, Datasets

TL;DR: Provides a novel, substantial foundation model dataset focused on multivariate time series, evaluation of state of the art models and analysis to identify correlation.

Abstract: Multivariate Time Series Foundation Models (TSFM) aim to identify patterns in multiple contexts to make meaningful predictions of the future. At their core is multivariate capability; models make use of information from multiple sources rather than relying on a single signal with limited information. Learning multivariate models requires meaningful evaluations, but current benchmarks are limited in two key ways: quantity and quality. There are a limited number of multivariate time series datasets, with existing ones lacking size and diversity across domains. Furthermore, although some collections of time series might be marketed as multivariate, it is not proven that they contain meaningful information in multiple contexts. This work takes a major step in both directions, providing a Multivariate Time Series Evaluation Dataset for Foundation Models (MUSE-FM). MUSE-FM spans 16 multivariate time series domains and introduces novel synthetic data techniques, comprising 67 billion data points and 2.6 million time series. To improve and prove the quality of multivariate data, we provide a powerful suite of benchmarking tools focused on identifying the multivariate predictability of a time series and introduce novel multivariate predictability aggregate metrics based on classical methods. Finally, we evaluate current state-of-the-art TSFM for both univariate and multivariate capability, finding that despite multivariate predictability identifying correlation, univariate prediction often matches or outperforms multivariate prediction across models.

Primary Area: learning on time series and dynamical systems

Submission Number: 7871