Effective Learning with Node Perturbation in Multi-Layer Neural Networks
Keywords: efficient machine learning, optimization
TL;DR: Input decorrelation and a mathematical reformulation of the update rule can greatly increase the convergence speed of Node Perturbation
Abstract: Backpropagation (BP) remains the dominant and most successful method for training parameters of deep neural network models.
However, BP relies on two computationally distinct phases, does not provide a satisfactory explanation of biological learning, and can be challenging to apply for training of networks with discontinuities or noisy node dynamics.
By comparison, node perturbation (NP) proposes learning by the injection of noise into network activations, and subsequent measurement of the induced loss change. NP relies on two forward (inference) passes, does not make use of network derivatives, and has been proposed as a model for learning in biological systems.
However, standard NP is highly data inefficient and unstable due to its unguided noise-based search process.
In this work, we investigate different formulations of NP and relate it to the concept of directional derivatives as well as combining it with a decorrelating mechanism for layer-wise inputs.
We find that a closer alignment with directional derivatives together with input decorrelation at every layer strongly enhances performance of NP learning with large improvements in parameter convergence and much higher performance on the test data, approaching that of BP.
Furthermore, our novel formulation allows for application to noisy systems in which the noise process itself is inaccessible.
Primary Area: optimization
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 10238
Loading