Prioritized Experience Replay for Continual Learning

Guannan Hu, Wu Zhang, Hu Ding, Wenhao Zhu

Student First Author: Yes

Keywords: Continual Learning, Tiny Episodic Memory, Prioritized Experience, Catastrophic Forgetting

Abstract: Human can learn and accumulate knowledge throughout their lifespan. Similarly, the paradigm of continual learning (CL) in artificial intelligence requires that the machine learning model can preserve consolidated knowledge if new task is adapted. However, to overcome catastrophic forgetting, a destructive issue in continual learning, memory-based approaches, replaying old experiences with experience drawn from novel task or constraining on old experiences, need a large memory to prevent them from decreasing consolidated knowledge, it is inefficiency and impractical. To improve the efficiency of old experiences and keep memory small, we introduce prioritized experience replay, which uses feature margin and classification margin to prioritize representative experiences. Feature margin is a cosine similarity between original experience and average experience, and classification margin is the correctness of model to predict experience. Experiment results show that prioritized experiences have a positive impact on alleviating catastrophic forgetting, and replaying prioritized experiences stored in tiny reservoir relieves over-fitting and outperforms state-of-the-art continual learning approaches in a training pass.

TL;DR: To overcome catastrophic forgetting in continual learning, we introduce prioritized experience replay with memory writing mechanism to relieve over-fitting and alleviate catastrophic forgetting.