UNDERSTANDING TRANSFORMERS FOR TIME SEIRES FORECASTING: A CASE STUDY ON MOIRAI

ICLR 2026 Conference Submission20122 Authors

Keywords: time series forecasting

TL;DR: We study transformers for time series forecasting, with a focus on MOIRAI.

Abstract:

We give a comprehensive theoretical analysis of transformers as time series pre-
diction models, with a focus on MOIRAI (Woo et al., 2024). We study its ap-
proximation and generalization capabilities. First, we demonstrate that there exist
transformers that fit an autoregressive model on input univariate time series via
gradient descent. We then analyze MOIRAI, one of the state-of-the-art multivariate
time series prediction models capable of modeling arbitrary number of covariates.
We prove that MOIRAI is capable of automatically fitting autoregressive models
with an arbitrary number of covariates, offering insights into its design and em-
pirical success. For generalization, we establish learning bounds for pretraining
when the data satisfies Dobrushin’s condition. Experiments support our theoretical
findings, highlighting the efficacy of using transformers for time series forecasting.

Primary Area: learning on time series and dynamical systems

Submission Number: 20122