UnHiPPO: Uncertainty-aware Initialization for State Space Models

Marten Lienen, Abdullah Saydemir, Stephan Günnemann

TL;DR: HiPPO extension based on linear stochastic control theory and the Kalman filter making SSMs more robust against noise

Abstract: State space models are emerging as a dominant model class for sequence problems with many relying on the HiPPO framework to initialize their dynamics. However, HiPPO fundamentally assumes data to be noise-free; an assumption often violated in practice. We extend the HiPPO theory with measurement noise and derive an uncertainty-aware initialization for state space model dynamics. In our analysis, we interpret HiPPO as a linear stochastic control problem where the data enters as a noise-free control signal. We then reformulate the problem so that the data become noisy outputs of a latent system and arrive at an alternative dynamics initialization that infers the posterior of this latent system from the data without increasing runtime. Our experiments show that our initialization improves the resistance of state-space models to noise both at training and inference time.

Lay Summary:

Many modern AI systems that understand time series—like audio or sensor data—rely on mathematical tools called state space models (SSMs). These models condense long sequences of data into a compact summary, which makes them fast and efficient. A popular approach, known as HiPPO, lets SSMs efficiently track patterns in data. However, HiPPO assumes that the input data is perfectly clean, without any random “noise” - which is rarely true for real-world data.

Our work extends HiPPO to handle noisy, imperfect data. By rethinking the math that underlies HiPPO, we create a new version called UnHiPPO that automatically filters out noise when summarizing data sequences. This makes state space models more robust and reliable—for example, when processing speech or sensor readings with background disturbances.

We show that this new approach improves how well models work on noisy data, both during training and when making predictions, on the example of LSSL. We hope our findings make powerful sequence models more useful and reliable for real-world applications where data is never perfectly clean.

Link To Code: https://cs.cit.tum.de/daml/unhippo

Primary Area: Deep Learning->Sequential Models, Time series

Keywords: state space, uncertainty, hippo, mamba, kalman, noise, filter

Submission Number: 16431