Zeroth-Order Forward-Only SNN Training Inspiring Neuromorphic On-Chip Learning

Mingyue Qin, Shuyu Yin, Qinghai Guo, Xiaolin Huang, Peilin Liu, Fei Wen

Keywords: Spiking neural networks, zeroth-order optimization

Abstract: The human brain is a biologically instantiated on-device neural system that integrates both learning and inference in a unified architecture, which enables rapid and flexible learning on-the-fly. This extraordinary capability is achieved through non-backpropagation learning mechanisms, whereas backpropagation (BP) is computationally and memory intensive which makes it unsuitable for on-chip edge learning. zeroth-order (ZO) optimization methods, which resemble biologically plausible perturbation-based learning, offer a promising alternative that enables learning with only forward passes and hence can significantly reduce the complexity of on-chip hardware implementation. However, in this work we show that applying ZO methods to spiking neural networks (SNNs) is non-trivial due to the step-function nature of spiking activation (e.g., Heaviside function). We analyze the challenges posed by the step-function activation, and propose a novel subspace-based zeroth-order (SZO) learning method that leverages the intrinsic low-dimensional structure of the SNN optimization trajectory. By learning in a low-dimensional subspace, SZO substantially enhances ZO learning efficacy, achieving accuracy comparable to first-order (FO) methods with faster learning speed than full-space BP. We evaluate SZO on model training from scratch, continual training, and unsupervised adaptation. Experimental results demonstrate that SZO closely approaches FO training performance for the first time while offering fast learning speed. We expect this work to inspire future research on highly efficient and scalable algorithms for neuromorphic on-chip learning.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 24108