There are no Champions in Long-Term Time Series Forecasting

Lorenzo Brigato, Rafael Morand, Knut Joar Strømmen, Maria Panagiotou, Markus Schmidt, Stavroula Mougiakakou

Keywords: Long-Term Time Series Forecasting, Benchmarking, Evaluation, Deep Learning, Transformers

TL;DR: We argue for a shift in Long-Term Time Series Forecasting from pursuing ever-more complex models and towards enhancing benchmarking practices through rigorous and standardized evaluation methods

Abstract:

Recent advances in long-term time series forecasting have introduced numerous complex prediction models that consistently outperform previously published architectures.
However, this rapid progression raises concerns regarding inconsistent benchmarking and reporting practices, which may undermine the reliability of these comparisons.
Our position emphasizes the need to shift focus away from pursuing ever-more complex models and towards enhancing benchmarking practices through rigorous and standardized evaluation methods.
To support our claim, we first perform a broad, thorough, and reproducible evaluation of the top-performing models on the most popular benchmark by evaluating five models over 14 datasets encompassing 3,500+ trained networks for the hyperparameter (HP) searches.
Then, through a comprehensive analysis, we find that slight changes to experimental setups or current evaluation metrics drastically shift the common belief that newly published results are advancing the state of the art.
Our findings suggest the need for rigorous and standardized evaluation methods that enable more substantiated claims, including reproducible HP setups and statistical testing.

Submission Number: 399