Sketched Gaussian Mechanism on Matrix for Private Federated LoRA

ICLR 2026 Conference Submission22249 Authors

Keywords: Federated Learning, Differential Privacy, Sketching, Communication Efficiency, LoRA

Abstract:

Low-Rank Adaptation (LoRA), which modifies frozen pre-trained parameters via the product of two trainable low-rank factors, has been widely adopted for communication-efficient fine-tuning of language models, including extensions to federated learning (FL). Nevertheless, two challenges arise at scale: (i) for very large models, the adapter factors can remain high-dimensional, leading to nontrivial communication costs between clients and the server; and (ii) transmitting local adapters between clients and the server risks privacy leakage. Incorporating differential privacy (DP) by additive mechanisms, e.g., the Gaussian mechanism (GM), often leads to substantial noise amplification, particularly in algorithms that must perturb both low-rank components.

In this paper, we propose the Sketched Gaussian Mechanism on Matrix (SGMM), which couples random sketching with the Gaussian mechanism at the matrix level. Using tools from Rényi differential privacy (RDP), we provide a unified analysis of SGMM’s privacy guarantee and show that, for a fixed privacy level, the required noise magnitude scales as $1/\sqrt{b}$ for sketch dimension $b$. Consequently, for moderate $b$, SGMM attains the same privacy with markedly less noise than GM. We instantiate SGMM within federated LoRA algorithms, including FFA-LoRA and FlexLoRA, where sketching further reduces adapter dimensionality and, in turn, the noise needed to meet a given privacy target, addressing both communication overhead and noise amplification. Experiments demonstrate that, at matched privacy budgets, SGMM-based federated LoRA is at least competitive with and in some settings outperforms non-sketched private baselines.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 22249