Go to ICLR 2024 Conference homepageRethinking the Starting Point: Enhancing Performance and Fairness of Federated Learning via Collaborative Pre-Training
Most existing federated learning (FL) methodologies have been developed starting from a randomly initialized model. Recently, several studies have empirically demonstrated that leveraging a pre-trained model can offer advantageous initializations for FL. In this paper, we take a departure from the assumption of centralized pre-training and instead focus on a practical FL setting, where data samples are distributed among both clients and the server even during the pre-training phase. We propose a collaborative pre-training approach for FL (CoPreFL), where the goal is to strategically design a pre-trained model that effectively serves as a good initialization for any downstream FL tasks. The key idea of our pre-training algorithm is to employ meta-learning to simulate downstream distributed scenarios, enabling it to adapt to unforeseen FL tasks. During optimization, CoPreFL also strikes a balance between average performance and fairness, with the aim of addressing the challenges in downstream FL tasks through initialization. Extensive experimental results validate that our pre-training method provides a robust initialization for any unseen downstream FL tasks, resulting in enhanced average performance and more equitable predictions.