Memory Efficient Continual Learning with TransformersDownload PDF

Published: 31 Oct 2022, Last Modified: 19 Jan 2023NeurIPS 2022 AcceptReaders: Everyone
Keywords: continual learning, transformers, ViT, BERT, text classification, image classification, adapters
TL;DR: In this work we devise a memory-efficient method to perform continual learning using pre-trained transformers models for text and images on a sequence of classification tasks.
Abstract: In many real-world scenarios, data to train machine learning models becomes available over time. Unfortunately, these models struggle to continually learn new concepts without forgetting what has been learnt in the past. This phenomenon is known as catastrophic forgetting and it is difficult to prevent due to practical constraints. For instance, the amount of data that can be stored or the computational resources that can be used might be limited. Moreover, applications increasingly rely on large pre-trained neural networks, such as pre-trained Transformers, since compute or data might not be available in sufficiently large quantities to practitioners to train from scratch. In this paper, we devise a method to incrementally train a model on a sequence of tasks using pre-trained Transformers and extending them with Adapters. Different than the existing approaches, our method is able to scale to a large number of tasks without significant overhead and allows sharing information across tasks. On both image and text classification tasks, we empirically demonstrate that our method maintains a good predictive performance without retraining the model or increasing the number of model parameters over time. The resulting model is also significantly faster at inference time compared to Adapter-based state-of-the-art methods.
Supplementary Material: pdf
45 Replies