Breaking Information Impedance in Deep Spiking Neural Networks via Multi-Stage Foundation-Model Distillation

lidongyang, Wei Wang, Qinghai Guo, Peilin Liu, Fei Wen

Keywords: Spiking Neural Networks, Information Impedance, Knowledge Distillation

Abstract: Brain-inspired spiking neural networks (SNNs) hold great promise for low-power, event-driven computation. Yet, their performance is fundamentally constrained by information impedance induced by spiking activations and spike-based propagation, a challenge that becomes more severe in deeper architectures and under limited time-steps. In this work, we conduct an information-theoretic analysis to reveal that such information impedance constitutes a key bottleneck to the learning of deep SNNs. To address it, we propose a multi-stage knowledge distillation (KD) method that leverages a high-capacity teacher model (DINOv2) to enhance the information extraction and transmission capabilities of SNNs. By decomposing a deep high-impedance path into low-impedance stages, our method effectively mitigates the representational bottlenecks caused by spike quantization. Extensive experiments demonstrate that our method substantially boost the learning of deep residual SNNs, e.g., on ImageNet-1K with ResNet-101, our method achieves 77.14\% top-1 accuracy, which surpasses the prior SOTA by 2.93\%. The gains are particularly significant for fully-spiking SNNs and deeper models. Importantly, while vanilla KD has been shown sufficient for ANNs on large-scale datasets, we show that for SNNs it is far from sufficient, and overcoming the information impedance is essential to fully unlock the potential of SNN distillation.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 24106