Emergence of Surprise and Predictive Signals from Local Contrastive Learning

Andrew Lawrence Smith, Linxing Preston Jiang, Stefano Recanatesi, Matthew Storm Bull

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Forward Forward Algorithm, Contrastive Learning, Predictive Coding, Cortical Representations, Biological Plausibility

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Hierarchical predictive models are often used to model cortical representations. These models exploit the local or global computation of predictive signals in the neural network, but their biological plausibility is limited as it is currently unknown whether cortical circuits perform such computations at all. This paper seeks to further investigate the inverted Forward-Forward Algorithm, a biologically plausible innovative approach to learning with only forward passes, in order to demonstrate that hierarchical predictive computations can emerge from a simpler contrastive constraint on the network's representation. Through the identification of compelling similarities between our model and hierarchical predictive coding, as well as the examination of the emergent properties of resulting representations, we advance the hypothesis that the computational properties that emerge in neocortical circuits, widely acknowledged as the basis of human intelligence, may be attributed to local learning principles.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8109