Federated Learning on Small Batch Sizes via Batch Renormalization

Rubing Xue, Jiaming Pei, Lukun Wang

Keywords: Federated Learning, Non-IID, Batch Renormalization

Abstract: When batch size is too small in personalized federated learning, the parameters of the model change significantly due to data variability and outliers, leading to highly stochastic gradients between the layers of the model. This situation ultimately extends the convergence time and poses a significant challenge to the model training process. To address this issue, we propose a method that involves local batch re-normalization before averaging the model. Experimental results demonstrate that this approach effectively improves the accuracy of personalized models trained with small batches in Non-IID scenarios.

Submission Number: 88