BP-Modified Local Loss for Efficient Training of Deep Neural Networks

REN Lianhai, Qianxiao Li

Keywords: deep learning optimization, local loss training, bias-variance balance

TL;DR: We proposed a novel method that periodically compute the BP gradient and use it to modify the local loss gradient. This method improve the performance of the original local loss methods with negligible additional memory usage.

Abstract:

The training of large models is memory-constrained, one direction to relieve
this is training using local loss, like GIM, LoCo, and Forward-Forward
algorithms. However, the local loss methods often face the issue of slow or
non-convergence. In this paper, we propose a novel BP-modified local loss
method that uses the true Backward Propagation (BP) gradient to modify the
local loss gradient to improve the performance of local loss training. We
use the stochastic modified equation to analyze our method and show that
modified offset decreases the bias between the BP gradient and local loss
gradient, but introduces additional variance, which results in a
bias-variance balance. Numerical experiments on full-tuning and LoKr tuning
on the ResNet-50 model and LoRA tuning on the ViT-b16 model on CIFAR-100
datasets show 20.5\% test top-1 accuracy improvement for the Forward-Forward
algorithm, 18.6\% improvement for LoCo algorithm and achieve only on average
7.7\% of test accuracy loss compared to the BP algorithm, with up to 75\%
memory savings.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1740