Stochastic Forward-Forward Learning through Representational Dimensionality Compression

Zhichao Zhu, YANG QI, Hengyuan Ma, Wenlian Lu, Jianfeng Feng

Keywords: Forward-Forward Algorithm, Effective Dimensionality, Energy-Based Learning, Neural Coding, Representational Geometry

TL;DR: We propose a second-order goodness function based on effective dimensionality for Forward-Forward learning, eliminating the need for negative samples and showing that noise enhances performance and inference robustness.

Abstract:

The Forward-Forward (FF) learning algorithm provides a bottom-up alternative to backpropagation (BP) for training neural networks, relying on a layer-wise "goodness" function with well-designed negative samples for contrastive learning.
Existing goodness functions are typically defined as the sum of squared postsynaptic activations, neglecting correlated variability between neurons. In this work, we propose a novel goodness function termed dimensionality compression that uses the effective dimensionality (ED) of fluctuating neural responses to incorporate second-order statistical structure. Our objective minimizes ED for noisy copies of individual inputs while maximizing it across the sample distribution, promoting structured representations without the need to prepare negative samples. We demonstrate that this formulation achieves competitive performance compared to other non-BP methods. Moreover, we show that noise plays a constructive role that can enhance generalization and improve inference when predictions are derived from the mean of squared output, which is equivalent to making predictions based on an energy term. Our findings contribute to the development of more biologically plausible learning algorithms and suggest a natural fit for neuromorphic computing, where stochasticity is a computational resource rather than a nuisance. The code is available at https://github.com/ZhichaoZhu/StochasticForwardForward.

Primary Area: General machine learning (supervised, unsupervised, online, active, etc.)

Submission Number: 2943