On the Necessity of Step-Specific Representation Learning for Multi-Step Forecasting

Licheng Pan, Zhijian Xu, Zi Ciu Chan, Haoxuan Li, Shuting He, Xiaoxi Li, Eric Wang, Yuan Lu, Qingsong Wen

Keywords: Time Series Forecasting, Mixture of Experts, Low Rank Adaptation, Partial Parameter Sharing

Abstract: This paper demonstrates that modern deep forecast models are susceptible to a fundamental **expressiveness bottleneck**, which stems from the use of step-invariant representations in multi-step prediction tasks. Through theoretical analysis, we demonstrate that step-invariant representation causes an unavoidable forecast error that cannot be overcome simply by advancing neural architectures. To address this issue, we propose Step-wise Representation adaPtation (SRP): first, a foundation model is pre-trained for one-step-ahead forecast; subsequently, the model is adapted to various forecast horizons via low-rank adaptation. This design enables the generation of step-specific representations, thereby avoiding the expressiveness bottleneck. Moving forward, we further establish SRP++, which employs adaptively weighted low-rank adapters to mitigate the expressiveness bottleneck while enhancing efficiency and forecast performance. Experiments show that SRP++ significantly improves model expressiveness and outperforms state-of-the-art time-series forecast methods. Code is available at~\url{https://anonymous.4open.science/r/SRP-7C55}.

Primary Area: learning on time series and dynamical systems

Submission Number: 23539