Heterogeneous Federated PEFTs via Sparse Mixture-of-Experts

ICLR 2026 Conference Submission21039 Authors

Keywords: Federated Learning, Personalization, LLMs, PEFT

TL;DR: We propose Hermes, a federated hybrid PEFT framework that unifies LoRA, Adapter, and Prefix modules with gradient-aware routing and balanced aggregation to enable efficient and personalized fine-tuning of large language models.

Abstract: Parameter-efficient fine-tuning (PEFT) has emerged as a critical technique for adapting large language models (LLMs) in federated learning (FL), enabling resource-efficient model updates without compromising user privacy. However, existing FL approaches predominantly rely on a single PEFT type shared across all clients, limiting their ability to handle the substantial data heterogeneity. In this work, we propose Hermes, a novel federated PEFT (FedPEFT) framework that introduces the concept of Heterogeneous FedPEFT, where each client flexibly combines multiple PEFT (e.g., LoRA, Adapter, Prefix Tuning) to better fit local data distributions. To address key challenges such as gradient conflicts, expert underutilization, and biased aggregation arising from this heterogeneous design, Hermes employs a structured sparse mixture-of-experts architecture with gradient-aware gating, loss-free bias adjustment, and inverse-frequency aggregation strategies. These techniques jointly ensure stable optimization and balanced contribution across clients. Extensive experiments on multiple NLP benchmarks demonstrate that Hermes achieves superior personalization performance compared to state-of-the-art homogeneous FedPEFT baselines, highlighting its potential as an effective solution for federated LLM fine-tuning under non-IID settings.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 21039