Curvature Explains Loss of Plasticity
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Plasitcity, neural networks, continual learning
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Loss of plasticity is a phenomenon in which neural networks lose their ability to learn from new experience.
Despite being empirically observed in several problem settings, little is understood about the mechanisms that lead to loss of plasticity.
In this paper, we offer a consistent explanation for plasticity loss,
based on an assertion that neural networks lose directions of curvature during training and that plasticity loss can be attributed to this reduction in curvature.
To support such a claim, we provide a systematic empirical investigation of plasticity loss across several continual supervised learning problems.
Our findings illustrate that curvature loss coincides with and sometimes precedes plasticity loss, while also showing that previous explanations are insufficient to explain loss of plasticity in all settings.
Lastly, we show that regularizers which mitigate loss of plasticity also preserve curvature, motivating a simple distributional regularizer that proves to be effective across the problem settings considered.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4057
Loading