Towards Interference-Free One-Shot Federated Learning via Frequency Domain Aggregation

Haoran Shi, Yifei Zhang, Hao Zhu, Conghao Xiong, Zengxiang Li, Irwin King, Piotr Koniusz, Han Yu

Keywords: ML

Abstract: The widespread adoption of Large Language Models (LLMs) has created new challenges for federated learning (FL), particularly due to their massive parameter counts that make traditional multi-round FL prohibitively expensive in terms of communication costs. One-shot federated learning (OFL), which limits communication to a single round, has emerged as a promising solution. However, existing OFL approaches suffer from performance degradation in the presence of heterogeneous data distributions, which limits their applicability in practice. To address this issue, we propose the one-shot \underline{Fed}erated \underline{F}requency \underline{Sep}arated aggregation (\methodname{}) method. It is a novel OFL framework for LLMs that leverages the discrete cosine transform (DCT) to construct orthogonal parameter spaces. This enables independent operations at FL client-side with minimal collisions, thereby facilitating effective model adaptation without iterative communications, even with heterogeneous data. By exploiting the natural orthogonality properties of DCT basis functions, our approach reduces the probability of interference among FL clients from $\mathcal{O}(|\Omega|^2/d)$ to $\mathcal{O}(|\Omega|^2/d^2)$, where $|\Omega|$ is the number of parameters updated per client and $d$ is the dimensionality of the model. Through theoretical analysis and extensive experiments on benchmark language understanding tasks, we demonstrate that \methodname{} outperforms existing one-shot methods while maintaining communication efficiency comparable to non-federated approaches

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 412