Spectral State Space Models

Naman Agarwal, Daniel Suo, Xinyi Chen, Elad Hazan

Keywords: deep learning, state space models, linear dynamical systems, sequence prediction

Abstract: This paper studies sequence modeling for prediction tasks with long range dependencies. We propose a new formulation for state space models (SSMs) based on learning linear dynamical systems with the spectral filtering algorithm \cite{hazan2017learning}. This gives rise to a novel sequence prediction architecture we call a spectral state space model. Spectral state space models have provable robustness properties for tasks that require long memory, and are constructed with fixed convolutional filters that do not need to be learned. We evaluate these models on synthetic dynamical systems and long-range prediction tasks of various modalities. These evaluations support the theoretical benefits of spectral filtering for tasks that need very long range memory.

Submission Number: 19