RFD-LoRA: Robust Federated Distillation for LoRA Fine-Tuning under Heterogeneous and Adversarial Clients

ICLR 2026 Conference Submission22509 Authors

Keywords: Federated Learning, LoRA Fine-Tuning, Knowledge Distillation, Adversarial Robustness

Abstract: Federated learning (FL) with low-rank adaptation (LoRA) is attractive for efficiency but fragile compared to full-rank FL. We show three fundamental vulnerabilities: (i) aggregation and projection bias, since bilinear averaging of adapters misrepresents the true global update; (ii) adversarial amplification, where low-rank projections can magnify malicious perturbations; and (iii) Jacobian sensitivity, where small adapter changes trigger large gradient variation. Existing methods only mitigate these issues and require identical client ranks, limiting practicality. We propose Robust Federated Distillation for LoRA (RFD-LoRA), the first framework to combine federated distillation with LoRA. By aggregating logits in a shared subspace, RFD-LoRA totally eliminates aggregation and initialization lag while enabling clients with heterogeneous ranks and adapter structures to collaborate seamlessly. To defend against non-IID and adversarial clients, we design three modules: Confidence-Adaptive Temperature (CAT), MMD-based Distillation (MMD-KD), and Disagreement Suppression (DIS). We provide error bounds and show on GLUE benchmarks that RFD-LoRA consistently outperforms prior methods in accuracy and robustness.

Primary Area: learning theory

Submission Number: 22509