FedMCP: Parameter-Efficient Federated Learning with Model-Contrastive Personalization

Anonymous

Abstract: Given the growing concerns over data privacy and security, fine-tuning pre-trained language models (PLMs) in federated learning (FL) has become the standard practice. However, this process faces two primary challenges. Firstly, the utilization of large-scale PLMs introduces excessive communication overheads. Secondly, the data heterogeneity across FL clients presents a major obstacle in achieving the desired fine-tuning performance. To address these challenges, we present a parameter-efficient fine-tuning (PEFT) method with Model-Contrastive Personalization (FedMCP). This approach introduces two adapter modules to the frozen PLM and only aggregates the global adapter in the federated aggregate phase while the private adapter stays in clients. The model-contrastive regularization term and aggregation strategy encourage the global adapter to learn universal knowledge from all clients and the private adapter to capture idiosyncratic knowledge for each individual client. Verified across a highly heterogeneous cross-silo dataset, the empirical evaluation shows considerable performance improvement achieved by FedMCP over state-of-the-art approaches.

Paper Type: long

Research Area: Machine Learning for NLP

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English