Federated Learning under Partially Disjoint Data via Manifold Reshaping

Ziqing Fan, Jiangchao Yao, Ruipeng Zhang, Lingjuan Lyu, Yanfeng Wang, Ya Zhang

Abstract: Statistical heterogeneity severely limits the performance of federated learning (FL), motivating several explorations e.g., FedProx, MOON and FedDyn, to alleviate this problem. Despite effectiveness, their considered scenario generally requires samples from almost all classes during the local training of each client, although some covariate shifts may exist among clients. In fact, the natural case of partially class-disjoint data (PCDD), where each client contributes a few classes (instead of all classes) of samples, is practical yet underexplored. Specifically, the unique collapse and invasion characteristics of PCDD can induce the biased optimization direction in local training, which prevents the efficiency of federated learning. To address this dilemma, we propose a manifold reshaping approach called FedMR to calibrate the feature space of local training. Our FedMR adds two interplaying losses to the vanilla federated learning: one is the intra-class loss to decorrelate feature dimensions for anti-collapse; and the other one is the inter-class loss to guarantee the proper margin among categories in the feature expansion. We conduct extensive experiments on a range of datasets to demonstrate that our FedMR achieves much higher accuracy and better communication efficiency.

Submission Length: Regular submission (no more than 12 pages of main content)

Code: https://github.com/MediaBrain-SJTU/FedMR.git

Assigned Action Editor: ~Virginia_Smith1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 1298