Hybrid Mamba–Transformer Decoder for Error-Correcting Codes

Shy-El Cohen, Yoni Choukroun, Eliya Nachmani

Keywords: Error-Correcting Codes, Mamba

TL;DR: A Mamba–Transformer hybrid error correcting codes decoder

Abstract: We introduce a novel deep learning method for decoding error correction codes based on the Mamba architecture, enhanced with Transformer layers. Our approach proposes a hybrid decoder that leverages Mamba’s efficient sequential modeling while maintaining the global context capabilities of Transformers. To further improve performance, we design a novel layer-wise masking strategy applied to each Mamba layer, allowing selective attention to relevant code features at different depths. Additionally, we introduce a progressive layer-wise loss, supervising the network at intermediate stages and promoting robust feature extraction throughout the decoding process. Comprehensive experiments across a range of linear codes demonstrate that our method significantly outperforms Transformer-only decoders and standard Mamba models.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 1978