A Reality Check on Pre-training for Exemplar-free Class-Incremental Learning

Eva Feillet, Adrian Popescu, Céline Hudelot

Abstract: Exemplar-free class-incremental learning (EFCIL) aims to classify streaming data without storing examples from the past. Recent EFCIL works suggest that (i) models pre-trained with large amounts of data should be used to initialize learning, (ii) self-supervised learned transformers generalize better than supervised convolutional models, (iii) adding generated data to the pre-training dataset can improve incremental accuracy. In this article, we question the above assertions by comprehensively evaluating various initial training strategies combined with four EFCIL algorithms using four large-scale datasets. Our results indicate that: (i) Pre-trained models are preferable when the domain of the incremental classification task is well represented in the pre-training datasets, but training with initial data remains useful when the domain shift is significant, (ii) supervised convolutional networks remain competitive, particularly when improving representation transferability using data augmentation or a projector, (iii) adding classes from an external dataset to train the initial model boosts performance when the initial set of classes is small but has a limited effect otherwise, (iv) additional classes generated with a diffusion model are not necessarily more useful than a well-chosen set of ImageNet classes to improve model transferability. We provide a nuanced analysis of these results and formulate recommendations to facilitate the practical adoption of EFCIL algorithms.