SaFARi: State-Space Models for Frame-Agnostic Representation

TMLR Paper5222 Authors

Abstract:

State-Space Models (SSMs) have re-emerged as a powerful tool for online function approximation, and as the backbone of machine learning models for long-range dependent data.
However, to date, only a few polynomial bases have been explored for this purpose, and the state-of-the-art implementations were built upon the best of a few limited options. In this paper, we present a generalized method for building an SSM with any frame or basis, rather than being restricted to polynomials.
This framework encompasses the approach known as HiPPO, but also permits an infinite diversity of other possible "species" within the SSM architecture. We dub this approach SaFARi: SSMs for Frame-Agnostic Representation.

Submission Length: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=SlCMxNfB1W

Changes Since Last Submission:

This is the third revision of our manuscript. For ease of reference, changes and additions after the first round of reviews are still in blue, and the newest edits are in purple. The main changes from the previous version are as follows:

1) The planned experimental results in Section 7 are complete, and additional discussion has been added where appropriate.
2) Section 7.1.2 (function approximation) now includes a discussion of rank vs. effective rank for SSMs constructed from redundant frames. As our paper advocates for moving beyond polynomial bases to handle cases where closed-form solutions do not exist, addressing redundancy and rank deficiency adds important context. Two additional entries in Table 3 are included for comparison of full-rank and rank-deficient SSMs in the experimental section.
3) Table 4 has been replaced by Figure 9. Section 7.1.3 (the delay/copying experiment) has been extended to include a much more densely sampled set of delays, which were better represented as a plot than a table.
4) A summary of key takeaways has been added to the conclusion, which we hope will clarify and crystallize the main messages of our paper for readers.
5) Miscellaneous edits for clarity have been made as recommended by Reviewer LaW1.

After making the additions and edits described above, we have made some efforts to reduce the length of the paper. Some content, such as the descriptions of the functions and polynomial families for the experiments, has been moved to the appendix. Figures and tables have been slightly resized or relocated to optimize for use of space. The main paper content is still 17 pages, the same as our previous version.

Assigned Action Editor: ~Grigorios_Chrysos1

Submission Number: 5222