Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation

Yehjin Shin, Seojin Kim, Noseong Park

Keywords: Mamba, State State Model, Graph Signal Processing

TL;DR: HADES reinterprets Mamba2 as a graph-based adaptive filter bank, achieving efficient and interpretable sequence modeling with fewer parameters.

Abstract: State-space models (SSMs) provide efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, employs selective input gating and a multi-head structure for parallel computation and strong performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we introduce \textbf{H}ierarchical \textbf{AD}aptive filter bank for \textbf{E}fficient \textbf{S}SMs (\textit{HADES}), a GSP-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. \textit{HADES} features two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved via structured bias on parameter $\Delta$. \textit{HADES} matches baseline models, including Mamba2, on key benchmark tasks while using only \textbf{58.9\%} of the original parameters.

Submission Number: 84