PhiNets: Brain-inspired Non-contrastive Learning Based on Temporal Prediction Hypothesis
Keywords: Non contrastive learning, Predictive coding, Eigenvalue dynamics, Temporal prediction hypothesis
TL;DR: We introduce PhiNet, a model based on the temporal prediction hypothesis, which outperforms SimSiam in stability and adaptability within non-contrastive learning.
Abstract: Predictive coding has been established as a promising neuroscientific theory to describe the mechanism of long-term memory residing in the retina.
This theory hypothesises that cortex predicts sensory inputs at various levels of abstraction to minimise prediction errors and forms long-term memory.
Inspired by predictive coding, Chen et al. (2024) proposed another theory, temporal prediction hypothesis, to claim that sequence memory residing in hippocampus has emerged through predicting input signals from the past sensory inputs.
Specifically, they supposed that the CA3 predictor in hippocampus creates synaptic delay between input signals, which is compensated by the following CA1 predictor.
Though recorded neural activities were replicated based on the temporal prediction hypothesis, its validity has not been fully explored.
In this work, we aim to explore the temporal prediction hypothesis from the perspective of self-supervised learning (SSL).
Specifically, we focus on non-contrastive learning, which generates two augmented views of an input image and predicts one from another.
Non-contrastive learning is intimately related to the temporal prediction hypothesis because the synaptic delay is implicitly created by StopGradient.
Building upon a popular non-contrastive learner, SimSiam, we propose PhiNet, an extension of SimSiam to have two predictors explicitly corresponding to the CA3 and CA1, respectively.
Through studying the PhiNet model, we discover two findings.
First, meaningful data representations emerge in PhiNet more stably than in SimSiam.
This is initially supported by our learning dynamics analysis: PhiNet is more robust to the representational collapse.
Second, PhiNet adapts more quickly to newly incoming patterns in online and continual learning scenarios.
For practitioners, we additionally propose an extension called X-PhiNet integrated with a momentum encoder, excelling in continual learning.
All in all, our work reveals that the temporal prediction hypothesis is a reasonable model in terms of the robustness and adaptivity.
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 7245
Loading