FSSM: Frequency-Selective State Space Models for Spectral Representation Learning

Anuvab Sen, Mir Sayeed Mohammad, Saibal Mukhopadhyay

Keywords: frequency spectrum, state space models, spectral operators, band-selective kernels, radar object detection, audio keyword recognition, sequence models

TL;DR: We propose a state-space model for learning spectral representations of time series data inspired by discrete Fourier transforms.

Abstract: We introduce the first state space model (FSSM) with frequency selective spectral operators, parameterizing a family of stable, causal, band-selective kernels whose spectral weights are conditioned on the end task. This yields a representation that adapts its characteristics per task domain while retaining linear-time inference and memory. The key novelty is the trainable spectral front-end through which the model can adapt frequency weighting and inter-bin window size. We show the effectiveness of our learned spectral representations on two independent domains: radar object detection and speech keyword recognition, outperforming state of the art frequency based methods in both domains while maintaining competitive throughput and computational overhead. We further show the robustness of our approach under input perturbations, demonstrating the value of stabilized sequential operators in spectral representation learning.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 23270