EFFEKT: Efficient Federated Knowledge Transfer to Foundation Models

TMLR Paper8962 Authors

Abstract:

Recent data protection laws have accelerated the adoption of Federated Learning (FL) for privacy-preserving decentralized training. Nevertheless, increasing model sizes imposes substantial computational demands on client devices, limiting FL applicability in resource-
constrained settings. We introduce a novel multi-domain federated learning framework in which lightweight client-side proxy models collaborate with a server-side Foundation Model (FM) to learn new concepts without sharing private data. Our approach, EFFEKT, enables
efficient server-side training of domain-specific LoRA adapters while preserving feature-space alignment between the FM and proxy extractors via novel bi-directional cross-distillation strategies. Experiments on multiple real-world datasets and deployments on low-power edge devices demonstrate consistent improvements over state-of-the-art baselines while maintaining lightweight computation at client side.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Hsuan-Tien_Lin1

Submission Number: 8962