Noise Transforms Feed-Forward Networks into Sparse Coding Networks

Trenton Bricken, Bruno Olshausen, Gabriel Kreiman

Keywords: Sparse Coding, Sparsity, Top-K Activation, Noise, Biologically Inspired

TL;DR: We find that noise alone induces networks to become Top-K, sparse coding networks. This resolves a difference between biological and artificial neural networks with regards to how sparse they are and how this sparsity is implemented..

Abstract: A hallmark of biological neural networks, which distinguishes them from their artificial counterparts, is the high degree of sparsity in their activations. Here, we show that by simply injecting symmetric, random, noise during training in reconstruction or classification tasks, artificial neural networks with ReLU activation functions eliminate this difference; the neurons converge to a sparse coding solution where only a small fraction are active for any input. The resulting network learns receptive fields like those of primary visual cortex and remains sparse even when noise is removed in later stages of learning.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Neuroscience and Cognitive Science (e.g., neural coding, brain-computer interfaces)