HotFed: Hot Start through Self-Supervised Learning in Federated Learning

Juan Zhao, Ruixuan Li, Haozhao Wang, Zijun Xu

Abstract: Federated Learning learns a shared model for prediction from the rich and privacy-sensitive training data distributed on the clients which are resource-constrained edge devices or isolated cloud clusters with high-security protection. The training data collected by separated clients faces the non-IID (not Independent and Identically Distributed) challenge, which requires more communication rounds to achieve an acceptable precision and hence costs considerable computing and network resources. The bias of the dataset requires more communication rounds to achieve an acceptable precision which costs a lot of computing and network resources. Existing works propose sharing a small amount of dataset among clients to mitigate the non-IID issue, bringing a new communication burden for shared data transition and the new distribution statistical challenge that the model will inevitably over-fit the shared data. Inspired by the hot start-up in system design, this paper designs a novel HotFed method in which each client starts its training with a representation model for better precision and communication efficiency. This paper analyzes the value of HotFed and different methods to warm up the client's Federated Learning. The conclusion turns out that HotFed through Self-Supervised Learning (SSL) can efficiently solve the non-IID issue and receive the training efficiency improvement.