A comprehensive, application-oriented study of catastrophic forgetting in DNNs

B. Pfülb, A. Gepperth

Abstract:

We present a large-scale empirical study of catastrophic forgetting (CF) in modern Deep Neural Network (DNN) models that perform sequential (or: incremental) learning.
A new experimental protocol is proposed that takes into account typical constraints encountered in application scenarios.
As the investigation is empirical, we evaluate CF behavior on the hitherto largest number of visual classification datasets, from each of which we construct a representative number of Sequential Learning Tasks (SLTs) in close alignment to previous works on CF.
Our results clearly indicate that there is no model that avoids CF for all investigated datasets and SLTs under application conditions. We conclude with a discussion of potential solutions and workarounds to CF, notably for the EWC and IMM models.

Keywords: incremental learning, deep neural networks, catatrophic forgetting, sequential learning

TL;DR: We check DNN models for catastrophic forgetting using a new evaluation scheme that reflects typical application conditions, with surprising results.

Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10), [Fashion-MNIST](https://paperswithcode.com/dataset/fashion-mnist), [MNIST](https://paperswithcode.com/dataset/mnist), [SVHN](https://paperswithcode.com/dataset/svhn)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1905.08101/code)