DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical Representations

Guogang Zhu, Xuefeng Liu, Jianwei Niu, Shaojie Tang, Xinghao Wu, Jiayuan Zhang

Abstract:

In personalized federated learning (PFL), it is widely recognized that achieving both high model generalization and effective personalization poses a significant challenge due to their conflicting nature. As a result, existing PFL methods can only manage a trade-off between these two objectives. This raises an interesting question: Is it feasible to develop a model capable of achieving both objectives simultaneously? Our paper presents an affirmative answer, and the key lies in the observation that deep models inherently exhibit hierarchical architectures, which produce representations with various levels of generalization and personalization at different stages.
A straightforward approach stemming from this observation is to select multiple representations from these layers and combine them to concurrently achieve generalization and personalization. However, the number of candidate representations is commonly huge, which makes this method infeasible due to high computational costs. To address this problem, we propose DualFed, a new method that can directly yield dual representations correspond to generalization and personalization respectively, thereby simplifying the optimization task.
Specifically, DualFed inserts a personalized projection network between the encoder and classifier. The pre-projection representations are able to capture generalized information shareable across clients, and the post-projection representations are effective to capture task-specific information on local clients. This design minimizes the mutual interference between generalization and personalization, thereby achieving a win-win situation. Extensive experiments show that DualFed can outperform other FL methods.

Primary Subject Area: [Systems] Systems and Middleware

Relevance To Conference: In the submitted manuscript, we focus on addressing the challenge of statistical heterogeneity in federated learning. Federated learning has became a pivotal technique in the realm of multimedia and multimodal processing, partically in scenarios where the data acquisition nodes are distributed across various locations. By enabling decentralized training of models on multiple devices without requiring the multimedia data to leave its source, federated learning respects user privacy while harnessing diverse and rich data. Our research delves into a cross-domain scenario where each client involved in the training has data from various domains, a common situation in multimedia processing applications such as autonomous driving and video surveillance. Our proposed method utilizes representations at different stages during the representation extraction to fulfill the dual requirements of generalization and personalization in federated learning. The experimental results on multiple datasets demonstrate that our method can achieve significant improvements in model performance compared to other federated learning approaches. In conclusion, our work holds potential to advance the application of federated learning in multimedia processing, particularly in contexts where data is sourced from diverse domains.

Submission Number: 3136