Communication-Efficient Federated Learning via Model-Agnostic Projection Adaptation

Mohammad Mahdi Rahimi, Younghyun Park, Humaira Kousar, Hasnain Irshad Bhatti, Dong-Jun Han, Jaekyun Moon

Keywords: Federated Learning, Low-Rank Adaptation, Communication Efficiency, Subspace Optimization

Abstract:

Federated learning (FL) enables collaborative model training across distributed clients without centralizing sensitive raw data while benefiting from diverse data sources.
Despite recent advancements in FL, the communication overhead remains a significant challenge, especially for large-scale models.
Recent low-rank adaptation (LoRA) techniques have shown promise in reducing these burdens in FL, but they are typically applied to each layer individually and depend on the model architecture, which limits their performance.
To address these shortcomings, we propose Model-Agnostic Projection Adaptation (MAPA), a novel approach that applies factorization to the entire model parameter space, which we view as a *single vector*, regardless of the number of layers and model architecture.
MAPA factorizes the single-vector model update into a fixed *reconstruction matrix* and a trainable *projection vector*, with the reconstruction matrix being randomly initialized using a shared seed at each round.
This ensures that *only* the projection vectors need to be communicated to the server, thereby reducing the communication cost.
Furthermore, MAPA's vector-based representation and relaxed rank constraints allow for a larger reconstruction matrix and smaller projection vector dimensions compared to LoRA, enhancing the expressiveness of model updates while significantly reducing communication overhead.
Experimental results demonstrate that MAPA outperforms existing FL methods in both communication efficiency and model performance, effectively coupling optimization and communication efficiency in FL environments.

Primary Area: other topics in machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11275