Local K-Similarity Constraint for Federated Learning with Label Noise

TMLR Paper5425 Authors

Abstract:

Federated learning on clients with noisy labels is a challenging problem, as such clients can infiltrate the global model, impacting the overall generalizability of the system. Existing methods proposed to handle noisy clients assume that a sufficient number of clients with clean labels are available, which can be leveraged to learn a robust global model while dampening the impact of noisy clients. This assumption fails when a high number of heterogeneous clients contain noisy labels, making the existing approaches ineffective. In such scenarios, it is important to locally regularize the clients before communication with the global model, to ensure the global model isn't corrupted by noisy clients. While pre-trained self-supervised models can be effective for local regularization, existing centralized approaches relying on pretrained initialization are impractical in a federated setting due to the large size of these models. In that line, we propose a regularization objective for client models that decouples the pre-trained and classification models by enforcing similarity between close data points within the client.
We leverage the representation space of a self-supervised pretrained model to evaluate the closeness among examples. This regularization, when applied with standard objective function for the downstream task in standard noisy federated settings, significantly improves performance, outperforming existing state-of-the-art federated methods in multiple computer vision and medical image classification benchmarks. Unlike other techniques that rely on self-supervised pretrained initialization, our method does not require the pretrained model and classifier backbone to share the same architecture, making it architecture-agnostic.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

Based on reviewer's comments and suggestions, we've made several changes to the manuscript:
- We have added a literature review section on Contrastive Objective in Federated Learning.
- We have made some edits to the methodology section for improved clarity based on reviewer's suggestions and comments.
- Based on reviewer 2's extensive suggestions, we have made changes to make notation more accurate and precise.
- We have corrected grammatical inaccuracies in the manuscript.
- In the appendix section, we have added section A.2 to discuss dependency of our approach on SSL model. We have also added section A.3 to discuss computation overhead of our approach.
- We have changed the location of the algorithm from appendix to methodology section (Section 3)

We've highlighted major changes in red.

Assigned Action Editor: ~Jean_Honorio1

Submission Number: 5425