WaLRUS: Wavelets for Long range Representation Using State Space Methods

Hossein Babaei, Mel White, Sina Alemohammad, Richard Baraniuk

Keywords: State-space models, SSM, HiPPO, Mamba, wavelet, safari, time series, sequence modeling, online function approximation, online memory, +

TL;DR: We introduce WaLRUS, a wavelet-based SSM leveraging SaFARi for improved accuracy and stability in modeling non-smooth, transient signals, outperforming traditional HiPPO-based models.

Abstract:

State-Space Models (SSMs) have proven to be powerful tools for online function approximation and for modeling long-range dependencies in sequential data. While recent methods such as HiPPO have demonstrated strong performance using a few polynomial bases, they remain limited by their reliance on closed-form solutions for specific, well-behaved bases.
The SaFARi framework generalizes this approach, enabling the construction of SSMs from arbitrary frames, including non-orthogonal and redundant ones, thus allowing an infinite diversity of possible "species'' within the SSM family. In this paper, we introduce WaLRUS (Wavelets for Long-range Representation Using SSMs), a new species of SaFARi built from Daubechies wavelet frames. We instantiate two variants, scaled-Walrus and translated-Walrus, and show that their multiresolution and localized nature offers significant advantages in representing non-smooth and transient signals. We compare Walrus to HiPPO-based models and demonstrate improved accuracy, better numerical properties, and more efficient implementations for online function approximation tasks.

Primary Area: General machine learning (supervised, unsupervised, online, active, etc.)

Submission Number: 23010