MS-SSM: A Multi-Scale State Space Model for Enhanced Sequence Modeling

Mahdi Karami, Ali Behrouz, Peilin Zhong, Razvan Pascanu, Vahab Mirrokni

Keywords: State Space Models, Sequence Models

TL;DR: This paper introduces a multi-resolution SSM framework that addresses these limitations by representing sequence dynamics across multiple resolutions.

Abstract: State Space Models (SSMs) have emerged as a promising alternative to computationally expensive attention-based models for sequence modeling. They rely on linear recurrences to integrate information over time, which enables for fast inference while still allowing the model to be parallelized during training and to control the stability of the recurrence. However, a consequence is that the effective memory of traditional SSMs is limited, requiring larger state sizes for improved recall. This paper introduces a multi-resolution SSM framework that addresses these limitations by representing sequence dynamics across multiple levels of detail. This approach captures both fine-grained, high-frequency patterns and coarse, low-frequency trends, hence effectively capturing historical patterns at multiple time scales. This decompositions allow the SSM to make better use of its memory. Our multi-resolution SSM demonstrates superior performance in various sequence modeling tasks, particularly in domains where multi-resolution patterns naturally occur, such as time series analysis and image processing.

Submission Number: 84