Evaluating Online Continual Learning with CALM

Germán Kruszewski, Ionut Teodor Sorodoc, Tomas Mikolov

Keywords: online continual learning, catastrophic forgetting, benchmark, language modelling

Abstract: Online Continual Learning (OCL) studies learning over a continuous data stream without observing any single example more than once, a setting that is closer to the experience of humans and systems that must learn “on-the-wild”. Yet, commonly available benchmarks are far from these real world conditions, because they explicitly signal different tasks, lack latent similarity structure or assume temporal independence between different examples. Here, we propose a new benchmark for OCL based on language modelling in which input alternates between different languages and domains without any explicit delimitation. Additionally, we propose new metrics to study catastrophic forgetting in this setting and evaluate multiple baseline models based on compositions of experts. Finally, we introduce a simple gating technique that learns the latent similarities between different inputs, improving the performance of a Products of Experts model.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We introduce a benchmark for Online Continual Learning based on language modelling, evaluating multiple baselines and improving one of them.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=5AB5JfhcI