Enhancing Spiking Transformers with Binary Attention Mechanisms

Guobin Shen, Dongcheng Zhao, Sicheng Shen, Yi Zeng

Keywords: Attention Binarization, Adaptive Threshold, Spiking Neural Network

TL;DR: Our study enhances Spiking Neural Networks with a novel binary attention mechanism, improving efficiency and accuracy on image and neuromorphic datasets.

Abstract: Spiking Neural Networks (SNNs) are increasingly recognized as an efficient alternative to traditional artificial neural networks. Recent advancements, particularly the integration of SNNs with Transformer structures to create 'SpikFormer', have significantly enhanced the performance of SNNs. However, the current non-spiking form of attention in SpikFormer poses risks of attention value explosion and still results in high computational costs for SNNs. To address this issue, we propose a novel binary attention mechanism. By introducing an attention shift mechanism and adaptive thresholds for neurons, we have successfully binarized the attention matrices in SpikFormer, leading to more efficient and sparser spiking neural networks. Experiments on image and neuromorphic datasets demonstrate that our approach maintains comparable performance to the original SpikFormer while reducing computational costs.

Submission Number: 40