Stream: A Generalized Continual Learning Benchmark and Baseline
Keywords: Continual Learning, Generalization, Multi-modal
Abstract: In a typical Continual Learning (CL) setting, the goal is to learn a sequence of tasks that are presented once while maintaining performance on all previously learned tasks. Current state-of-the-art approaches require the task identity during training to mitigate forgetting, whereas General Continual Learning (GCL) evaluates the ability to learn the sequence of tasks without their identity. We find that GCL methods ignore the domain gap between two consecutive tasks (‘learning-gap’) and, as a result, often fail under more challenging scenarios. Motivated by a learner that needs to generalize across modalities and tasks, we propose a challenging GCL benchmark: the multi-modal Stream. Our benchmark provides a method to construct a sequence of tasks with varying learning-gaps from Vision and Text datasets. We perform a systematic analysis of meta-training statistics from the literature that are used to identify novel tasks, to find that they correlate to the learning-gap. Inspired by biological mechanisms of learning in mammals, we propose a baseline method to achieve GCL on Stream: αMetaSup, which uses a ‘dummy’ Stream to train a Transformer model to identify novel task transitions (‘surprise’). The trained Transformer is then used as an auxiliary novelty detector to a learner in the benchmark Stream. We show how αMetaSup can augment existing CL methods that use rehearsal memory and improve their performance by as much as 10.5% AUC thereby outperforming 7 SOTA GCL baselines.
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6080
Loading