GIFF: Generalized Inference Friendly Forward-Forward Algorithm

Yigit Tuncel, Jiahao Lin, Xinmiao Xiong, Umit Ogras

Keywords: Forward-Forward algorithm, on-device learning, tinyML, edge computing, machine learning, memory efficient

TL;DR: This paper presents the Generalized Inference-Friendly Forward-Forward (GIFF) algorithm to address the three major limitations of the recently published Forward-Forward algorithm.

Abstract: The Forward-Forward (FF) algorithm has recently been proposed to enable neural network training using only forward passes, inspired by the human cortex excitation and inhibition mechanisms. In contrast to Backpropagation (BP), which uses a global loss function, FF utilizes local loss functions at each layer, reducing peak memory requirements. The local weight update scope allows alternative optimizers, non-differentiable neural network layers, and aggressive quantization. Despite its promise for on-device training, the original FF technique has three major limitations that hinder its potential: the label embedding problem, lack of support for convolutional layers, and inefficient inference passes. These issues hamper its performance even on basic datasets like CIFAR10 and restrict its applicability. This paper presents the Generalized Inference Friendly Forward-Forward (GIFF) algorithm to address the limitations of the FF algorithm. We demonstrate GIFF on three representative tinyML benchmarks where FF cannot function. GIFF performs as well as BP on all three tasks, using up to 43% less memory. Furthermore, GIFF requires significantly fewer computations than FF for inference. Thus, GIFF unlocks the potential benefits of the FF algorithm for efficient on-device learning.

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7625