Emergent representations in networks trained with the Forward-Forward algorithm
Abstract: The Backpropagation algorithm has often been criticised for its lack of biological realism. In an attempt to find a more biologically plausible alternative, the recently introduced Forward-Forward algorithm replaces the forward and backward passes of Backpropagation with two forward passes. In this work, we show that the internal representations obtained by the Forward-Forward algorithm can organise into category-specific ensembles exhibiting high sparsity -- composed of a low number of active units. This situation is reminiscent of what has been observed in cortical sensory areas, where neuronal ensembles are suggested to serve as the functional building blocks for perception and action. Interestingly, while this sparse pattern does not typically arise in models trained with standard Backpropagation, it can emerge in networks trained with Backpropagation on the same objective proposed for the Forward-Forward algorithm.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: Changes in Discussion and Appendix for rebuttal.
Main new experiments:
- goodness L1
- comparison with regularised BP
- representation similarity analysis
- linear probes for decoding information in representations of unseen categories
Major revisions in the "Discussion and conclusion" section and addition of statistical tests on ensembles
Code: https://github.com/NiccoloTosato/EmergentRepresentations
Assigned Action Editor: ~William_T_Redman1
Submission Number: 3735
Loading