Local Superior Soups: A Catalyst for Reducing Communication Rounds in Federated Learning with Pre-trained Model

Minghui Chen, Meirui Jiang, Qi Dou, Zehua Wang, Xiaoxiao Li

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Federated Learning; Model Soup; Pre-Trained Model Fine-Tuning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A better local model soup can accelerate federated fine-tuning

Abstract:

Federated learning (FL) is a learning paradigm that enables collaborative training of models using decentralized data.
Recently, the utilization of pre-trained weight initialization in FL has been demonstrated to effectively improve model performance.
However, the current pre-trained models have become increasingly parameter-rich.
The sheer scale of model parameters introduces substantial communication rounds challenges during their adaptation to FL.
To address these communication cost issues and elevate the performance of pre-trained model adaptation in FL, we propose an innovative model interpolation-based local training technique called ``Local Superior Soups.''
Our method promotes local training across different clients, encouraging the exploration of a connected low-loss basin within a few communication rounds through regularized model interpolation.
This approach serves as a facilitator for pre-trained model adaptation in FL.
We demonstrated its effectiveness and efficiency across diverse widely-used FL datasets.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1432