Energy-Based Models for Continual Learning

Shuang Li, Yilun Du, Gido Martijn van de Ven, Antonio Torralba, Igor Mordatch

Keywords: Continual learning, Energy-based model

Abstract: We motivate Energy-Based Models (EBMs) as a promising model class for continual learning problems. Instead of tackling continual learning via the use of external memory, growing models, or regularization, EBMs have a natural way to support a dynamically-growing number of tasks and classes and less interference with old tasks. We show that EBMs are adaptable to a more general continual learning setting where the data distribution changes without the notion of explicitly delineated tasks. We also find that EBMs outperform the baseline methods by a large margin on several continual learning benchmarks. These observations point towards EBMs as a class of models naturally inclined towards the continual learning regime.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We show Energy-Based Models are a class of models naturally inclined towards the continual learning regime.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2011.12216/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=VTkiOinmwC