Do Spiking Neural Networks Learn Similar Representation with Artificial Neural Networks? A Pilot Study on SNN Representation

Yuhang Li, Youngeun Kim, Hyoungseob Park, Priyadarshini Panda

Keywords: Spiking Neural Networks, Artificial Neural Network, Representation Similarity Analysis

TL;DR: Systematic study on the representation difference between ANNs and SNNs are conducted in this work.

Abstract: Spiking Neural Networks (SNNs) have recently driven much research interest owing to their bio-plausibility and energy efficiency. The biomimicry spatial-temporal communication and computation mechanisms are the key differences that set SNNs apart from current Artificial Neural Networks (ANNs). However, some essential questions exist pertaining to SNNs and yet are little studied: Do SNNs learn similar representation with ANN? Does the time dimension in spiking neurons provide additional information? In this paper, we aim to answer these questions by conducting a representation similarity analysis between SNNs and ANNs using Centered Kernel Alignment~(CKA). We start by analyzing the spatial dimension of the networks, including both the width and the depth. Furthermore, our analysis of residual connection shows that SNN learns a periodic pattern, which rectifies the representations in SNN to ANN-like. We additionally investigate the effect of the time dimension on SNN representation, finding that deeper layers encourage more dynamics along the time dimension. Other aspects like potential improvement in terms of accuracy, efficiency, and adversarial robustness are also analyzed using CKA. We hope this work will inspire future research to fully comprehend the representation of SNNs.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Neuroscience and Cognitive Science (e.g., neural coding, brain-computer interfaces)