Distribution-Aware Multi-Granularity Phase Coding: Towards Lower Conversion Error for Spike-Driven Large Language Models

Hanyuan Zheng, Haozhen Zhang, Tianshuo Chen, Zhaogeng Liu, Yi Chang, Bin Gu

Keywords: Spiking Neural Network, Optimization

Abstract:

Spiking large language models (LLMs) offer significant advantages on neuromorphic hardware, yet training them from scratch remains prohibitively expensive. A promising alternative is ANN-to-SNN conversion, which reuses pretrained ANN weights while minimizing conversion error.
However, existing conversion frameworks neglect activation distributions, as reflected in SNN neurons with rate or temporal coding to map uniformly distributed rather than distribution-aligned discrete values, thus causing latent conversion error arising from distribution misalignment.
To tackle this problem, we propose a distribution-aware multi-granularity phase coding approach, which achieves reasonable discrete value allocation by minimizing conversion error relative to activation distributions.
Specifically, multi-granularity phase coding extends conventional phase coding with multiple learnable bases, incorporating representational capacity across different granularities.
Building on this coding scheme, we further propose a novel ANN-to-SNN conversion paradigm designed towards lower conversion error.
In particular, our paradigm utilizes the activation distributions of hidden layers to sample data for cost-efficient neuron training, without requiring fine-tuning of model weights.
Theoretically, we provide a convergence guarantee for the neuron training algorithm.
Extensive experiments on the LLaMA model confirm the effectiveness of both our coding scheme and conversion paradigm.
Concretely, our spiking LLM attains the lowest perplexity with ANN-level accuracy, accompanied by a 42\% reduction in energy consumption of MAC and AC operations.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 9103