Provable Length Generalization in Sequence Prediction via Spectral Filtering

Evan Dogariu, Annie Marsden, Naman Agarwal, Xinyi Chen, Daniel Suo, Peter Bartlett, Elad Hazan

Keywords: online learning, state space models, linear dynamical systems, sequence prediction, LLMs

TL;DR: We design a new spectral filtering algorithm that provably length generalizes and include proof-of-concept experiments.

Abstract: We consider the problem of length generalization in sequence prediction. We define a new metric of performance in this setting -- the Asymmetric-Regret -- which measures regret against a benchmark predictor with longer context length than available to the learner. We continue by studying this concept from the lens of the spectral filtering algorithm. We present a gradient-based learning algorithm that provably achieves length generalization for linear dynamical systems. We conclude with proof-of-concept experiments which are consistent with our theory.

Primary Area: learning on time series and dynamical systems

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7574