Continual Learning via Winning Subnetworks That Arise Through Stochastic Local Competition
Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Continual Learning, Winning Tickets, Stochastic Local Competition
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: This work addresses catastrophic forgetting in class-incremental learning, by considering winning subnetworks with stochastic task winner-takes-all (TWTA) activations.
Abstract: The aim of this work is to address catastrophic forgetting in class-incremental learning. To this end, we propose deep networks that comprise blocks of units that compete locally to win the representation of each arising new task; competition takes place in a stochastic manner. This type of network organization results in sparse task-specific representations from each network layer; the sparsity pattern is obtained during training and is different among tasks. Under this rationale, our continual task learning algorithm regulates gradient-driven weight updates for each unit in a block on the grounds of winning probability. During inference, the network retains only the winning unit and zeroes-out all weights pertaining to non-winning units for the task at hand. As we empirically show, our method produces state-of-the-art predictive accuracy on few-shot image classification experiments, and imposes a considerably lower computational overhead compared to the current state-of-the-art.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
Supplementary Material: zip
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 196
Loading