Leveraging State Space Models in Long Range Genomics
Track: Full Paper Track
Keywords: State Space Models, SSMs, long-range genomics, genomics, DNA modeling, zero-shot extrapolation, extrapolation, long-range dependencies, BioAI, sequence model
TL;DR: We show SSMs match transformers on long-range genomics, can zero-shot extrapolate to sequences 100× longer than those in training, and process sequences up to 1M tokens on a single GPU, making them an efficient and scalable solution for the same.
Abstract: Long-range dependencies are critical for understanding genomic structure and function, yet most conventional methods struggle with them. Widely adopted transformer-based models, while excelling at short-context tasks, are limited by the attention module's quadratic computational complexity and inability to extrapolate to sequences longer than those seen in training. In this work, we explore State-Space Models (SSMs) as a promising alternative by benchmarking two SSM inspired architectures, Caduceus and Hawk, on long-range genomics modeling tasks under conditions parallel to a 50M-parameter transformer baseline. We discover that SSMs match transformer performance and exhibit impressive zero-shot extrapolation across multiple tasks, handling contexts 10–100$\times$ longer than those seen during training, indicating more generalizable representations better suited for modeling the long and complex human genome. Moreover, we demonstrate that these models can efficiently process sequences of 1M tokens on a single GPU, paving the way for modeling entire genomic regions at once, even in labs with limited compute. Our findings establish SSMs as efficient and scalable for long-context genomic analysis.
Attendance: Anirudha Ramesh
Submission Number: 19
Loading