Efficient Message-Passing Transformer for Error Correcting Codes

ICLR 2026 Conference Submission23820 Authors

Keywords: Channel coding, Error correcting codes, Transformer-based decoder, Message-passing decoder, Neural decoder, Transformer, Efficient attention module

TL;DR: We propose a novel efficient message-passing decoder for error correcting codes based on the proposed efficient error correcting attention module.

Abstract: Error correcting codes (ECCs) are a fundamental technique for ensuring reliable communication over noisy channels. Recent advances in deep learning have enabled transformer-based decoders to achieve state-of-the-art performance on short codes; however, their computational complexity remains significantly higher than that of classical decoders due to the attention mechanism. To address this challenge, we propose EfficientMPT, an efficient message-passing transformer that significantly reduces computational complexity while preserving decoding performance. A key feature of EfficientMPT is the Efficient Error Correcting (EEC) attention mechanism, which replaces expensive matrix multiplications with lightweight vector-based element-wise operations. Unlike standard attention, EEC attention relies only on query-key interaction using global query vector, efficiently encode global contextual information for ECC decoding. Furthermore, EfficientMPT can serve as a foundation model, capable of decoding various code classes and long codes by fine-tuning. In particular, EfficientMPT achieves 85% and 91% of significant memory reduction and 47% and 57% of FLOPs reduction compared to ECCT for $(648,540)$ and $(1056,880)$ standard LDPC code, respectively.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 23820