There are no Champions among Supervised Long-Term Time Series Forecasting Models

TMLR Paper6051 Authors

Abstract:

Recent advances in long-term time series forecasting have introduced numerous complex supervised prediction models that consistently outperform previously published architectures.
However, this rapid progression raises concerns regarding inconsistent benchmarking and reporting practices, which may undermine the reliability of these comparisons.
In this study, we first perform a broad, thorough, and reproducible evaluation of the top-performing supervised models on the most popular benchmark and additional baselines representing the most active architecture families. This extensive evaluation assesses eight models on 14 datasets, encompassing $\sim$5,000 trained networks for the hyperparameter (HP) searches.
Then, through a comprehensive analysis, we find that slight changes to experimental setups or current evaluation metrics drastically shift the common belief that newly published results are advancing the state of the art.
Our findings emphasize the need to shift focus away from pursuing ever-more complex models, towards enhancing benchmarking practices through rigorous and standardized evaluation methods that enable more substantiated claims, including reproducible HP setups and statistical testing.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Devendra_Singh_Dhami1

Submission Number: 6051