Mimetic Initialization Helps State Space Models Learn to Recall

Asher Trockman, Hrayr Harutyunyan, J Zico Kolter, Sanjiv Kumar, Srinadh Bhojanapalli

Track: long paper (up to 8 pages)

Keywords: Mamba, SSM, sequence models, language models, initialization, linear attention

TL;DR: Mamba learns to recall more easily when we initialize it to be more like linear attention

Submission Number: 23