Continual Learning Without Knowing Task Identities: Do Simple Models Work?

Tiffany Tuor, Shiqiang Wang, Kin Leung

Keywords: continual learning, catastrophic forgetting, Bayesian neural network

Abstract: Due to the catastrophic forgetting phenomenon of deep neural networks (DNNs), models trained in standard ways tend to forget what it has learned from previous tasks, especially when the new task is sufficiently different from the previous ones. To overcome this issue, various continual learning techniques have been developed in recent years, which, however, often suffer from a substantially increased model complexity and training time. In this paper, we consider whether properly tailored simple models could perform well for continual learning. By proposing a relatively simple method based on Bayesian neural networks and model selection, we can in many cases outperform several state-of-the-art techniques in terms of accuracy, model size, and running time, especially when each mini-batch of data is known to come from the same task of an unknown identity. This leads to interesting observations suggesting that different continual learning techniques may be beneficial for different types of data and task diversity.

One-sentence Summary: We present a simple method for continual learning and evaluate its performance against various state-of-the-art techniques in terms of accuracy, model size, and computation time.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=KZ5X9AVhY