FedSSI: Rehearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence

Yichen Li, Yuying Wang, Haozhao Wang, Yining Qi, Tianzhe Xiao, Ruixuan Li

Abstract: Continual Federated Learning (CFL) allows distributed devices to collaboratively learn novel concepts from continuously shifting training data while avoiding \textit{knowledge forgetting} of previously seen tasks. To tackle this challenge, most current CFL approaches rely on extensive rehearsal of previous data. Despite effectiveness, rehearsal comes at a cost to memory, and it may also violate data privacy. Considering these, we seek to apply regularization techniques to CFL by considering their cost-efficient properties that do not require sample caching or rehearsal. Specifically, we first apply traditional regularization techniques to CFL and observe that existing regularization techniques, especially synaptic intelligence, can achieve promising results under homogeneous data distribution but fail when the data is heterogeneous. Based on this observation, we propose a simple yet effective regularization algorithm for CFL named \textbf{FedSSI}, which tailors the synaptic intelligence for the CFL with heterogeneous data settings. FedSSI can not only reduce computational overhead without rehearsal but also address the data heterogeneity issue. Extensive experiments show that FedSSI achieves superior performance compared to state-of-the-art methods.

Lay Summary: Continual Federated Learning (CFL) enables devices to collaboratively learn new knowledge without sharing data while retaining previous knowledge. Current methods require repeatedly storing previous data, which is memory-intensive and poses privacy risks. To address this, we propose FedSSI, a lightweight solution that skips data storage and intelligently adjusts learning focus to tackle performance drops caused by data variations. Experiments confirm FedSSI outperforms existing approaches while saving resources and safeguarding privacy.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Primary Area: General Machine Learning

Keywords: Federated Learning, Continual Federated Learning, Data Heterogeneity

Submission Number: 8999