DIET-CP: Lightweight and Data Efficient Self Supervised Continued Pretraining

Bryan Rodas, Natalie Montesino, Jakob Ambsdorf, David Klindt, Randall Balestriero

Keywords: Self Supervised Learning, Continued Pretraining, Domain Adaptation

TL;DR: DIET-CP uses a simple instance discrimination objective to efficiently improve foundation model performance on new domains

Abstract: Continued pretraining offers a promising solution for adapting foundation models to a new target domain. However, in specialized domains, available datasets are often very small, limiting the applicability of SSL methods developed for large-scale pretraining, and making hyperparameter search infeasible. In addition, pretrained models are usually released as backbone-weights only, lacking important information to continue pretraining. We propose to bridge this gap with DIET-CP, a simple continued pretraining strategy, where any strong foundation model can be steered towards the new data distribution of interest. DIET-CP relies on a very simple objective, requires no labels, introduces no more hyperparameters than supervised finetuning. It is stable across data modalities and backbone choices, while providing a significant performance boost for state-of-the-art models such as DINOv3 using only 1000 images.

Submission Number: 96