Differentially Private Continual Learning using Pre-Trained Models

Marlon Tobaben, Marcus Klasson, Rui Li, Arno Solin, Antti Honkela

Keywords: differential privacy, pre-trained models, privacy-preserving machine learning, continual learning, image classification

Abstract: This work explores the intersection of continual learning (CL) and differential privacy (DP). Crucially, continual learning models must retain knowledge across tasks, but this conflicts with the differential privacy requirement of restricting individual samples to be memorised in the model. We propose using pre-trained models to address the trade-offs between privacy and performance in a continual learning setting. More specifically, we present necessary assumptions to enable privacy-preservation and propose combining pre-trained models with parameter-free classifiers and parameter-efficient adapters that are learned under differential privacy. Our experiments demonstrate their effectiveness and provide insights into balancing the competing demands of continual learning and privacy.

Submission Number: 11