Communication Efficient Federated Learning via Model-Agnostic Projection Adaptation

Mohammad Mahdi Rahimi, Younghyun Park, Humaira Kousar, Hasnain Irshad Bhatti, Dong-Jun Han, Jaekyun Moon

Abstract:

Federated learning (FL) enables collaborative model training across distributed clients without centralizing sensitive data. Despite recent advancements, communication overhead remains a major bottleneck, particularly for large-scale models.
Utilizing low-rank adaptation (LoRA) techniques can mitigate this challenge by decomposing each layer into a *reconstruction matrix* and a *projection matrix*, and transmitting either both matrices or only the projection matrix while keeping
the reconstruction matrix fixed. While effective, these techniques operate on individual layers, are architecture-dependent, and suffer from performance limitations due to their fixed reconstruction matrix.
We propose **Model-Agnostic Projection Adaptation (MAPA)**, a novel factorization approach that treats the entire model parameter space as a single matrix rather than decomposing layers independently. MAPA introduces round-wise randomization of the reconstruction matrix to avoid suboptimal solutions while flexibly balancing communication and accuracy.
MAPA also reduces the memory and computational overhead relative to LoRA, ensuring efficiency in both communication and computation when applied to federated learning.
Empirical results demonstrate the effectiveness of MAPA in various FL settings.

Primary Area: Deep Learning->Everything Else

Keywords: Federated Learning, Low-Rank Adaptation, Communication Efficiency, Subspace Optimization

Submission Number: 2961