Long Range Dependency Understanding in State Space Models

Srividya Ravikumar, Shweta Verma, Abhinav Anand, Mira Mezini

Keywords: Methods (probing, steering, causal interventions)

Other Keywords: State space models

TL;DR: The paper studies filter behavior of SSM kernels in various architectural configurations and evaluate the short and long range dependency understanding of these models.

Abstract: Although state-space models (SSMs) have demonstrated strong performance on long-sequence benchmarks, most research has emphasized predictive accuracy rather than interpretability. In this work, we present the first systematic kernel interpretability study of the SSM kernel trained on a real-world task. We present time and frequency domain analysis of the SSM kernel, and show that the long-range modeling capability of SSM varies significantly under different model architectures, affecting model performance. We assess the long and short range dependency understanding of the models through their filter behavior. For instance, SSM kernel can behave as low-pass, band-pass or high-pass filter. The insights from our analysis can guide the future work in designing better SSM based models.

Submission Number: 683