Recovering Selectivity with LTI State Space Operators for Portable Long-Context Inference

Minseon Gwak, N. Benjamin Erichson, PooGyeon Park

Keywords: Foundation Models, Mamba, State Space Models, Adaptive Inference, Efficient Inference

Abstract: Mamba has established selective state space models as powerful building blocks for long-context foundation models by replacing linear time-invariant (LTI) recurrences with input-dependent selective recurrences. However, efficient training of such recurrences often relies on specialized fused kernels, limiting portability across diverse hardware. We propose SelectLTI, a simple but effective block that augments an LTI recurrence with input-dependent modulation to recover selective behavior. The proposed block equips any LTI state-space layer with sigmoid-gated modulators, making it trainable and deployable without specialized fused kernels. We evaluate SelectLTI on Selective Copying, a task designed to isolate selective-memory behavior in sequence models. SelectLTI matches or exceeds S6, the selective state-space operator of Mamba, showing that selective behavior can be recovered from an LTI recurrence augmented with lightweight modulation using only standard primitives. We position SelectLTI as a portable building block for efficient long-context foundation models deployed across diverse compute environments.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 47