One-Shot Federated Aggregation of Generalized Embeddings for Edge Environments

Ripu Daman Singh Bankawat, Rashmi Singh, Shrihari Dumbre, Bikash Santra

Keywords: One shot federated learning, knowledge distillation, edge environments, differential privacy

TL;DR: A one-shot federated learning framework introducing a unique progressive knowledge distillation for edge environments.

Abstract: Federated learning (FL) enables collaborative training in decentralized data; however, traditional multi-round protocols incur high {communication} costs. One-shot FL (OFL) reduces {communication} cost by limiting interaction to a single round. Existing OFL methods often suffer from poor accuracy and heavy client-side computation, making them unsuitable for resource-constrained edge devices. This paper introduces a novel OFL framework, FedAGE (Federated Aggregation of Generalized Embeddings), in which clients transmit latent representations derived from a shared frozen encoder and a portion of the model instead of full weights. This design offloads expensive computation to the server, drastically reducing the client's overhead of computation, while retaining essential discriminative features within the shared embeddings. FedAGE transfers knowledge to the server through a progressive distillation framework, incorporating weighted soft labels, ensemble distillation, and knowledge mixing to mitigate catastrophic forgetting. Extensive experiments are carried out on the five benchmark datasets. The proposed FedAGE consistently outperforms OFL baselines, achieving up to {46.4\% higher accuracy}, under high heterogeneous partitioning. The experiments evident that FedAGE's performance is consistent across various levels of data heterogeneity. Also, our analysis shows at least a 66\% reduction in client-side computational overhead, measured in GigaFLOPs. These findings confirm FedAGE as a viable framework for federated learning, offering scalability and efficiency without compromising accuracy in edge settings. The source code for our FedAGE is available at https://anonymous.4open.science/r/FEDAGE-One-ShotFederatedLearning-832E.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 23696