A Weight Variation-Aware Training Method for Hardware Neuromorphic Chips
Keywords: hardware implementation, synaptic device, edge computing systems, neuro-inspired computing
Abstract: Hardware neuromorphic chips that mimic the biological nervous systems have recently attracted significant attention due to their ultra-low power and parallel computation. However, the inherent variability of nano-scale synaptic devices causes a weight perturbation and performance drop of neural networks. This paper proposes a training method to find weight with robustness to intrinsic device variability. A stochastic weight characteristic incurred by device inherent variability is considered during training. We investigate the impact of weight variation on both Spiking Neural Network (SNN) and standard Artificial Neural Network (ANN) with different architectures including fully connected, CNN, VGG, ResNet, MobileNet and EfficientNet on MNIST, N-MNIST, CIFAR-10, CIFAR-100, and ImageNet. Experimental results show that a weight variation-aware training method (WVAT) can dramatically minimize the performance drop on weight variability by exploring a flat loss landscape. On ferroelectric tunnel junctions (FTJ) devices, WVAT yields 78.01% accuracy of VGG-5 on CIFAR-10 for weight perturbations, while SGD scores 28.43%. Finally, WVAT is easy to implement on various architectures with little computational overhead.
Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5307
Loading