Positional Encoding for Spiking Transformers

Zijian Zhou, Malu Zhang, Yu Liang, Honglin Cao, Ammar Belatreche, Jieyuan Zhang, Wenjie Wei, Shuai Wang, Yang Yang, Haizhou Li

Keywords: Spiking Neural Networks, Spiking Self-attention, Spiking Transformers, Positional Encoding

Abstract:

Spiking Neural Networks (SNNs) offer superior energy efficiency compared to Artificial Neural Networks (ANNs). Recent Transformer-based SNNs have achieved promising performance by integrating spike-driven computation with Transformer architectures.
Positional information is essential in sequential tasks. However, existing positional encoding methods designed for ANNs cannot be directly applied to SNNs, as they interfere with the spike-driven computation paradigm, highlighting the need for SNN-specific solutions.
We propose Spiking Positional Encoding (SPE), a novel positional encoding specifically designed for Spiking Transformers that captures both absolute and relative positional information. Its key component is the Positional Encoding Leaky Integrate-and-Fire (PE-LIF) neuron layer, which encodes positional information directly into neuron thresholds. Through continuous spike firing and membrane potential reset processes, this positional information is effectively reflected in the emitted spikes while preserving the spike-driven computation paradigm.
Comprehensive experiments across seven datasets, including three time-series forecasting tasks and four natural language processing benchmarks, demonstrate that SPE consistently outperforms existing positional encoding methods and achieves state-of-the-art performance.
SPE provides a tailored positional encoding solution for Spiking Transformers, bridging the performance gap between ANNs and SNNs, thus advancing neuromorphic computing applications in sequential modeling tasks.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 5753