Decoupling General and Personalized Knowledge in Federated Learning via Additive and Low-rank Decomposition

Xinghao Wu, Xuefeng Liu, Jianwei Niu, Haolin Wang, Shaojie Tang, Guogang Zhu, Hao Su

Abstract: To address data heterogeneity, the key strategy of personalized Federated Learning (PFL) is to decouple general knowledge (shared among clients) and client-specific knowledge, as the latter can have a negative impact on collaboration if not removed. Existing PFL methods primarily adopt a parameter partitioning approach, where the parameters of a model are designated as one of two types: parameters shared with other clients to extract general knowledge and parameters retained locally to learn client-specific knowledge. However, as these two types of parameters are put together like a jigsaw puzzle into a single model during the training process, each parameter may simultaneously absorb both general and client-specific knowledge, thus struggling to separate the two types of knowledge effectively. In this paper, we introduce FedDecomp, a simple but effective PFL paradigm that employs parameter additive decomposition to address this issue. Instead of assigning each parameter of a model as either a shared or personalized one, FedDecomp decomposes each parameter into the sum of two parameters: a shared one and a personalized one, thus achieving a more thorough decoupling of shared and personalized knowledge compared to the parameter partitioning method. In addition, as we find that retaining local knowledge of specific clients requires much lower model capacity compared with general knowledge across all clients, we let the matrix containing personalized parameters be low rank during the training process. Moreover, a new alternating training strategy is proposed to further improve the performance. Experimental results across multiple datasets and varying degrees of data heterogeneity demonstrate that FedDecomp outperforms state-of-the-art methods up to 4.9\%.

Primary Subject Area: [Systems] Systems and Middleware

Relevance To Conference: Federated learning has been widely applied in the design of multimedia artificial intelligence systems, garnering widespread attention among researchers for its ability to enable collaborative learning across multiple devices without directly sharing raw data. Multimedia systems often involve processing heterogeneous data from various sources, such as images, audio, and text, making data heterogeneity a primary challenge for effective federated learning. To mitigate this issue, we have designed FedDecomp that employs additive parameter decomposition to effectively separate general knowledge shared across all clients from client-specific knowledge within the model training process. FedDecomp is particularly suited for multimedia/multimodal processing scenarios, as it allows the model to retain general knowledge common to all clients while also learning unique knowledge specific to each client or modality. We have validated the effectiveness of FedDecomp on multiple modal data types, including image and text, and the results demonstrate its significant advantage in handling data heterogeneity, with performance surpassing existing state-of-the-art methods across various datasets.

Submission Number: 4884