Continual pre-training mitigates forgetting in language and vision

Andrea Cossu, Antonio Carta, Lucia C. Passaro, Vincenzo Lomonaco, Tinne Tuytelaars, Davide Bacciu

Abstract: Highlights•Continual Pre-Training incrementally acquires knowledge from unstructured data streams.•Self-Supervised Continual Pre-Training effectively mitigates forgetting.•The representation drift is reduced by Self-Supervised Continual Pre-Training.•Performance on domain-specific tasks can be improved with a limited amount of data.