Backpropagation-Free Learning through Gradient Aligned Feedbacks

Paul Caillon, Erwan Fagnou, Blaise Delattre, Alexandre Allauzen

Keywords: backpropagation-free, direct feedback alignment, optimization, forward gradient

Abstract:

Deep neural networks heavily rely on the back-propagation algorithm for optimiza-
tion. Nevertheless, the global sequential transmission of gradients in the backward
pass inhibits its scalability. The Direct Feedback Alignment algorithm has been
proposed as a promising approach for parallel learning of deep neural networks,
relying on fixed random feedback weights to project the error on every layer in
a parallel manner. However, it notoriously fails to train networks that are really
deep and that include compulsory layers like convolutions and transformers. In this
paper, we show that alternatives to back-propagation may greatly benefit from local
and forward approximation of the gradient to better cope with the inherent and
constrained structure of such layers.

This directional approximation allows us to design a novel algorithm that updates the feedback weights called GrAPE (GRadient
Aligned Projected Error). A first set of experiments are carried out on image classi-
fication tasks with feedforward and convolutional architectures. The results show
important improvement in performance over other backpropagation-free algorithms,
narrowing the gap with backpropagation. More importantly, the method scales
to modern and deep architectures like AlexNet, VGG-16 and Transformer-based
language models where the performance gains are even more notable.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9865