TKIL: Tangent Kernel Optimization for Class Balanced Incremental Learning

Jinlin Xiang, Eli Shlizerman

Keywords: Incremental Learning

Abstract: When sequentially learning multiple tasks, deep neural networks tend to loose accuracy on tasks learned in the past while gaining accuracy on the current task. This phenomenon is called catastrophic forgetting. Class Incremental Learning (CIL) methods address this problem by keeping a memory of exemplars from previous tasks, which are supposed to assist with overall accuracy of all tasks. However, existing methods struggle to balance the accuracy across all seen tasks since there is still overfitting to the current task due to data imbalance between the complete training data points for the current task and limited exemplars in the memory buffer for previous tasks. Here, we propose to avoid the data imbalance by learning a set of generalized non-task-specific parameters. In particular, we propose a novel methodology of Tangent Kernel for Incremental Learning (TKIL) that seeks an equilibrium between current and previous representations. We achieve such equilibrium by computing and optimizing for a new Gradient Tangent Kernel (GTK). Specifically, TKIL tunes task-specific parameters for all tasks with GTK loss. Therefore, when representing previous tasks, task-specific models are not influenced by the samples of the current task and are able to retain learned representations. As a result, TKIL equally considers the contribution from all task models. The generalized parameters that TKIL obtains allow our method to automatically identify which task is being considered and to adapt to it during inference. Extensive experiments on 5 CIL benchmark datasets with 10 incremental learning settings show that TKIL outperforms existing state-of-the-art methods, e.g., a 9.4% boost on CIFAR100 with 25 incremental stages. Furthermore, TKIL attains strong state-of-the-art accuracy on the large-scale dataset, with a much smaller model size (36%) compared to other approaches.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning

TL;DR: Tangent Kernel optimization for class balanced Incremental Learning that addresses the imbalances in memory-based incremental learning.