S4++: Elevating Long Sequence Modeling with State Memory Reply

Biqing Qi, Junqi Gao, Dong Li, Kaiyan Zhang, Jianxing Liu, Ligang Wu, Bowen Zhou

Keywords: State Space model, Long Sequence Modeling

Abstract:

Recently, state space models (SSMs) have shown significant performance advantages in modeling long sequences.
However, in spite of their promising performance, there still exist limitations.
1) Non-Stable-States (NSS): Significant state variance discrepancies arise among discrete sampling steps, occasionally resulting in divergence.
2) Dependency Bias: The unidirectional state space dependency in SSM impedes the effective modeling of intricate dependencies.
In this paper, we conduct theoretical analysis of SSM from the even-triggered control (ETC) theory perspective and first propose the presence of NSS Phenomenon.

Our findings indicate that NSS primarily results from the sampling steps, and the integration of multi-state inputs into the current state significantly contributes to the mitigation of NSS.
Building upon these theoretical analyses and findings, we propose a simple, yet effective, theoretically grounded State Memory Reply (SMR) mechanism that leverages learnable memories to incorporate multi-state information into the current state. This enables the precise modeling of finer state dependencies within the SSM, resulting in the introduction of S4+.
Furthermore, we integrate the complex dependency bias into S4+ via interactive cross attentions mechanism, resulting in the development of S4++.
Our extensive experiments in autoregressive language modeling and benchmarking against the Long Range Arena demonstrate superior performance in most post-processing tasks.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7483