Reshape-then-Factorize: Communication-Efficient FL via Model-Agnostic Projection Optimization

ICLR 2026 Conference Submission13968 Authors

Keywords: Federated Learning, Low-Rank Adaptation, Communication Efficiency, Subspace Optimization

Abstract:

Federated learning (FL) enables collaborative model training across distributed clients without sharing sensitive data. However, communication overhead remains a significant bottleneck, particularly for large-scale models. Low-rank decomposition techniques address this by approximating each layer’s weights or gradients with a product of low-rank matrices, thereby reducing the communication cost in FL. While effective, these methods are constrained by the layer's architecture and shapes, limiting their flexibility and performance.
We propose *Model-Agnostic Projection Optimization* (MAPO), a novel method that reshapes and factorizes the full model gradient into a *fixed reconstruction matrix* and a *trainable projection vector*, avoiding layer-wise decomposition and architecture constraints. MAPO directly optimizes the projection in a randomly sampled subspace, with all clients generating the reconstruction matrix via a shared random seed, incurring no additional communication overhead for synchronization.
By decoupling the gradient from architectural constraints through reshaping and enabling communication-free exploration of dynamic subspaces via seed sharing, MAPO provides a more flexible and efficient low-rank representation.
Empirical results demonstrate the effectiveness of MAPO in various FL settings.

Primary Area: optimization

Submission Number: 13968