Accelerating Training of Deep Spiking Neural Networks with Parameter Initialization

Jianhao Ding, Jiyuan Zhang, Zhaofei Yu, Tiejun Huang

Keywords: spiking neural networks, back-propagationthrough time, parameter initialization

Abstract: Despite that spiking neural networks (SNNs) show strong advantages in information encoding, power consuming, and computational capability, the underdevelopment of supervised learning algorithms is still a hindrance for training SNN. Our consideration is that proper weight initialization is a pivotal issue for efficient SNN training. It greatly influences gradient generating with the method of back-propagation through time at the initial training stage. Focusing on the properties of spiking neurons, we first derive the asymptotic formula of their response curve approximating the actual neuron response distribution. Then, we propose an initialization method obtained from the slant asymptote to overcome gradient vanishing. Finally, experiments with different coding schemes on classification tasks show that our method can effectively improve training speed and the final model accuracy compared with traditional deep learning initialization methods and existing SNN initialization methods. Further validation on different neuron types and training hyper-parameters has shown comparably good versatility and superiority over the other methods. Some suggestions are given to SNN training based on the analyses.

One-sentence Summary: An SNN initialization method based on properties of spiking neuron response is proposed to accelerate training and get better accuracy compared with existing methods.