FedDUAL: A Dual-Strategy with Adaptive Loss and Dy- namic Aggregation for Mitigating Data Heterogeneity in Federated Learning

TMLR Paper5474 Authors

Abstract: Federated Learning (FL) marks a transformative approach to distributed model training by combining locally optimized models from various clients into a unified global model. While FL preserves data privacy by eliminating centralized storage, it encounters significant challenges such as performance degradation, slower convergence, and reduced robustness of the global model due to the heterogeneity in client data distributions. Among the various forms of data heterogeneity, label skew emerges as a particularly formidable and prevalent issue, especially in domains such as image classification. To address these challenges, we begin with comprehensive experiments to pinpoint the underlying issues in the FL training process. Based on our findings, we then introduce an innovative dual-strategy approach designed to effectively resolve these issues. First, we introduce an adaptive loss function for client-side training, meticulously crafted to preserve previously acquired knowledge while maintaining an optimal equilibrium between local optimization and global model coherence. Secondly, we develop a dynamic aggregation strategy for aggregating client models at the server. This approach adapts to each client's unique learning patterns, effectively addressing the challenges of diverse data across the network. Our comprehensive evaluation, conducted across three diverse real-world datasets, coupled with theoretical convergence guarantees, demonstrates the superior efficacy of our method compared to several established state-of-the-art approaches.

Submission Length: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=GGmdrvy7Vd

Changes Since Last Submission:

We thank the Action Editor for the opportunity to resubmit our work, and we are grateful to all reviewers for recognizing the novelty and contributions of our paper, as well as for their constructive feedback (anomalies in the theoretical convergence analysis). Their insights have been instrumental in improving the manuscript, and we have revised the paper accordingly to address all suggestions. We have updated both convex and non-convex convergence analysis in detail in the appendix. Below, we outline the key revisions made in response to the feedback:

Based on the suggestion of reviewer $\textbf{cNkU}$, we have thoroughly revisited the proofs, filled in the missing steps, and clarified the arguments for both convex and non-convex settings. A comprehensive, step-by-step derivation is now included in Appendix (Section A). We welcome any further feedback or suggestions for improvement. Following the reviewer $\textbf{aNcb}$ suggestion, we have conducted additional motivating experiments (see section 1.1 and Figure 6), and ablations across multiple datasets (see Table 3 and 4 for ablation experiments). Detailed explanations for “ for the lower gains compared to FedAvg in ablation studies” have also been added in Section 5.0.1, highlighting their impact and relevance. Based on the reviewer $\textbf{NWxU}$ suggestions, we have revised the manuscript accordingly: added graphs with confidence intervals (see Fig. 11), relocated the ablation studies to the main body for better visibility, and incorporated all recommended clarifications and writing improvements.

Assigned Action Editor: ~Haoliang_Li2

Submission Number: 5474