Distil-xLSTM: Learning Attention Mechanisms through Recurrent Structures

ACL ARR 2025 May Submission7788 Authors

Abstract:

The current era of Natural Language Processing (NLP) is dominated by Transformer models. However, novel architectures relying on recurrent mechanisms, such as xLSTM and Mamba, have been proposed as alternatives to attention-based models. Although computation is done
differently than with the attention mechanism, these recurrent models yield good results and sometimes even outperform state-of-the-art attention-based models. In this work, we propose Distil-xLSTM, an xLSTM-based Small Language Model (SLM) trained by distilling knowledge
from a Large Language Model (LLM) that shows promising results while being compute and scale efficient. Our Distil-xLSTM focuses on approximating a transformer-based model attention parametrization using its recurrent sequence mixing components and shows good results with minimal training.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: xLSTM, Transformer, Knowledge Distillation, LLM, SLM

Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 7788