Towards Understanding What State Space Models Learn About Code

ACL ARR 2026 May Submission14621 Authors

Keywords: State Space Models, Interpretability, Transformers, Code models

Abstract:

State Space Models (SSMs) have emerged as an efficient alternative to the Transformer architecture. Prior work shows that, when trained under comparable conditions, SSMs can match or surpass Transformers on code understanding tasks. However, their internal mechanisms remain a black box.
We present the first systematic analysis of what SSM-based code models learn along with the direct comparison between SSM and Transformer models in this domain. Our analysis shows that SSMs capture syntactic and semantic structure more effectively than Transformers during pretraining but forgets certain relations during fine-tuning on some tasks. To investigate this behavior, we introduce \textit{SSM-Interpret}, a frequency-domain framework that exposes a “spectral shift" toward short-range dependencies during fine-tuning. Guided by these findings, we propose architectural modifications that significantly improve the performance of SSM-based code model by upto +6 MRR on NLCodeSearch. This demonstrates that our analysis not only explains model behavior but also leads directly to better designs.

Paper Type: Long

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: Probing

Contribution Types: Model analysis & interpretability

Languages Studied: Python, English, TypeScript

EMNLP 2026 AI Reviewing Experiment: no

Submission Number: 14621