Fusion of Global and Local Knowledge for Personalized Federated Learning

Published: 11 Mar 2023, Last Modified: 11 Mar 2023Accepted by TMLREveryoneRevisionsBibTeX
Abstract: Personalized federated learning, as a variant of federated learning, trains customized models for clients using their heterogeneously distributed data. However, it is still inconclusive about how to design personalized models with better representation of shared global knowledge and personalized pattern. To bridge the gap, we in this paper explore personalized models with low-rank and sparse decomposition. Specifically, we employ proper regularization to extract a low-rank global knowledge representation (GKR), so as to distill global knowledge into a compact representation. Subsequently, we employ a sparse component over the obtained GKR to fuse the personalized pattern into the global knowledge. As a solution, we propose a two-stage proximal-based algorithm named \textbf{Fed}erated learning with mixed \textbf{S}parse and \textbf{L}ow-\textbf{R}ank representation (FedSLR) to efficiently search for the mixed models. Theoretically, under proper assumptions, we show that the GKR trained by FedSLR can at least sub-linearly converge to a stationary point of the regularized problem, and that the sparse component being fused can converge to its stationary point under proper settings. Extensive experiments also demonstrate the superior empirical performance of FedSLR. Moreover, FedSLR reduces the number of parameters, and lowers the down-link communication complexity, which are all desirable for federated learning algorithms. Source code is available in \url{https://github.com/huangtiansheng/fedslr}.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: * Re-organizing the text and layout for the aesthetic purpose. * Adding source code url in the abstract. * Adding acknowledgment after the main body. * Correcting typos.
Code: https://github.com/huangtiansheng/fedslr
Assigned Action Editor: ~Ruoyu_Sun1
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 479