Numerical Analysis of HiPPO-LegS ODE for Deep State Space Models

TMLR Paper6128 Authors

Abstract: In deep learning, the recently introduced state space models utilize HiPPO (High-order Polynomial Projection Operators) memory units to approximate continuous-time trajectories of input functions using ordinary differential equations (ODEs), and these techniques have shown empirical success in capturing long-range dependencies in long input sequences. However, the mathematical foundations of these ODEs, particularly the singular HiPPO-LegS (Legendre Scaled) ODE, and their corresponding numerical discretizations remain unsettled. In this work, we fill this gap by establishing that HiPPO-LegS ODE is well-posed despite its singularity, albeit without the freedom of arbitrary initial conditions. Further, we establish convergence of the associated numerical discretization schemes for Riemann integrable input functions.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

We have revised the manuscript to incorporate the feedback from the reviewers. The main changes are summarized below:

* **Presentation & Clarity:** Added a summary table of discretization schemes and their convergence guarantees in the Introduction, and included a new Conclusion section.

* **Numerical Experiments:** Added comparisons of finite-$n$ error constants across discretization schemes to strengthen the practical validation.

* **Theoretical Clarifications:** Expanded the proof details for Corollary 2 and Theorem 2, and added a brief finite-horizon stability argument in Remark 3.3.

* **Typos:** Corrected various typos, including ones that were pointed out by the reviewers.

Assigned Action Editor: ~Pierre_Ablin2

Submission Number: 6128