Exploiting Client Heterogeneity for Forgetting Mitigation in Federated Continual Learning: A Spatio-Temporal Gradient Alignment Approach

Duong Minh Nguyen, Le-Tuan Nguyen, Dung D. Le, Quoc-Viet Pham

Keywords: Federated Continual Learning, Gradient Matching, Heterogeneous task

TL;DR: This paper introduces a gradient matching approach to leverage client-specific knowledge to mitigate forgetting in heterogeneous federated continual learning

Abstract: Federated Continual Learning (FCL) has recently emerged as a crucial research area, as data from distributed clients typically arrives as a stream, requiring sequential learning. This paper explores a more practical and challenging FCL setting, where clients may have unrelated or even heterogeneous tasks, leading to gradient conflicts where local updates point in divergent directions. In such scenario, statistical heterogeneity and data noise can create spurious correlations, leading to biased feature learning and catastrophic forgetting. Existing FCL approaches often use generative replay to create pseudo-datasets of previous tasks. However, generative replay itself suffers from catastrophic forgetting and task divergence among clients, leading to overfitting in FCL. To address these challenges, we propose a novel approach called \textbf{\underline{S}}patio-\textbf{\underline{T}}emporal gr\textbf{\underline{A}}dient \textbf{\underline{M}}atching with \textbf{\underline{P}}rototypical Coreset (STAMP). Our contributions are threefold: 1) We develop a model-agnostic method to determine subset of samples that effectively form prototypes when using a prototypical network, making it resilient to continual learning challenges; 2) We introduce a spatio-temporal gradient matching approach, applied at both the client-side (temporal) and server-side (spatio), to mitigate catastrophic forgetting and data heterogeneity; 3) We leverage prototypes to approximate task-wise gradients, improving gradient matching on the client-side. Extensive experiments demonstrate our method's superiority over existing baselines, particularly in scenarios with a large number of sequential tasks, highlighting its effectiveness in addressing the complexities of real-world FCL.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 10955