RMAAT: A Bio-Inspired Approach for Efficient Long-Context Sequence Processing in Transformers

Md Zesun Ahmed Mia, Malyaban Bal, Abhronil Sengupta

Keywords: Astrocyte, Neuromorphic Computing, Bio Inspired Learning, Neuroscience-Algorithm-Application Codesign

TL;DR: RMAAT introduces a bio-inspired transformer model leveraging astrocytic memory to efficiently handle long-context sequences with reduced memory usage and enhanced computational speed

Abstract: Astrocytes, an essential component of the brain's neural circuitry, demonstrate learning capabilities through bioplausible mechanisms such as presynaptic plasticity and hebbian plasticity. However, their integration into computational models remains underexplored. This paper advances astromorphic computing techniques to emulate transformer self-attention mechanisms, leveraging astrocytic nonlinearity and memory retention to improve long-range dependency processing in machine learning and natural language processing (NLP) tasks. Existing transformer models have difficulty handling lengthy contexts with thousands of tokens, even with substantial computational resources. We propose Recurrent Memory Augmented Astromorphic Transformers (RMAAT), integrating astrocytic memory and recurrent processing into self-attention, enabling longer context handling without quadratic complexity growth. Our bioplausible model has been found to outperform traditional transformers in experimental tests conducted on the Long Range Arena benchmark and IMDB dataset. Specifically, our model achieves a significant reduction in memory utilization and computational latency. This paves the way for biologically inspired AI models by illustrating how astrocytic characteristics may enhance the performance and efficiency of computational models.

Primary Area: other topics in machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 12886