BP-Modified Local Loss for Efficient Training of Deep Neural Networks

ICLR 2025 Conference Submission1740 Authors

Keywords: deep learning optimization, local loss training, bias-variance balance

TL;DR: We proposed a novel method that periodically compute the BP gradient and use it to modify the local loss gradient. This method improve the performance of the original local loss methods with negligible additional memory usage.

Abstract:

The training of large models is memory-constrained, one direction to relieve
this is training using local loss, like GIM, LoCo, and Forward-Forward
algorithms. However, the local loss methods often face the issue of slow or
non-convergence. In this paper, we propose a novel BP-modified local loss
method that uses the true Backward Propagation (BP) gradient to modify the
local loss gradient to improve the performance of local loss training. We
use the stochastic modified equation to analyze our method and show that
modified offset decreases the bias between the BP gradient and local loss
gradient, but introduces additional variance, which results in a
bias-variance balance. Numerical experiments on full-tuning and LoKr tuning
on the ResNet-50 model and LoRA tuning on the ViT-b16 model on CIFAR-100
datasets show 20.5\% test top-1 accuracy improvement for the Forward-Forward
algorithm, 18.6\% improvement for LoCo algorithm and achieve only on average
7.7\% of test accuracy loss compared to the BP algorithm, with up to 75\%
memory savings.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1740