Continual Deep Learning by Functional Regularisation of Memorable Past

Pingbo Pan, Alexander Immer, Siddharth Swaroop, Runa Eschenhagen, Richard E Turner, Mohammad Emtiyaz Khan

TL;DR: This paper introduces a scalable functional-regularisation approach for continual learning that uses a GP formulation of neural networks to identify and regularise over a memorable past.

Abstract: Continually learning new skills without forgetting old ones is an important quality for an intelligent system, yet most deep learning methods suffer from catastrophic forgetting of the past. Recent works have addressed this by regularising the network weights, but it is challenging to identify weights crucial to avoid forgetting. A better approach is to directly regularise the network outputs at past inputs, e.g., by using Gaussian processes (GPs), but this is usually computationally challenging. In this paper, we propose a scalable functional-regularisation approach where we regularise only over a few memorable past examples that are crucial to avoid forgetting. Our key idea is to use a GP formulation of deep networks, enabling us to both identify the memorable past and regularise over them. Our method achieves state-of-the-art performance on standard benchmarks and opens a new direction for life-long learning where regularisation methods are naturally combined with memory-based methods.

Keywords: Continual learning, deep learning, functional regularisation

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2004.14070/code)