Pretraining with Random Noise for Fast and Robust Learning without Weight Transport

Jeonghwan Cheon, Sang Wan Lee, Se-Bum Paik

Keywords: Random noise training, Network pretraining, Pre-regularization, Feedback alignment, Error backpropagation, Weight transport problem, Biologically-Plausible Algorithm

TL;DR: Pretraining with random noise using a feedback alignment algorithm allows fast learning and robust generalization without weight transport.

Abstract: The brain prepares for learning even before interacting with the environment, by refining and optimizing its structures through spontaneous neural activity that resembles random noise. However, the mechanism of such a process has yet to be understood, and it is unclear whether this process can benefit the algorithm of machine learning. Here, we study this issue using a neural network with a feedback alignment algorithm, demonstrating that pretraining neural networks with random noise increases the learning efficiency as well as generalization abilities without weight transport. First, we found that random noise training modifies forward weights to match backward synaptic feedback, which is necessary for teaching errors by feedback alignment. As a result, a network with pre-aligned weights learns notably faster and reaches higher accuracy than a network without random noise training, even comparable to the backpropagation algorithm. We also found that the effective dimensionality of weights decreases in a network pretrained with random noise. This pre-regularization allows the network to learn simple solutions of a low rank, reducing the generalization error during subsequent training. This also enables the network to robustly generalize a novel, out-of-distribution dataset. Lastly, we confirmed that random noise pretraining reduces the amount of meta-loss, enhancing the network ability to adapt to various tasks. Overall, our results suggest that random noise training with feedback alignment offers a straightforward yet effective method of pretraining that facilitates quick and reliable learning without weight transport.

Primary Area: Neuroscience and cognitive science (neural coding, brain-computer interfaces)

Submission Number: 11053