BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach

Haozhao Wang, Shengyu Wang, Jiaming Li, Hao Ren, XINGSHUO HAN, Wenchao Xu, Shangwei Guo, Tianwei Zhang, Ruixuan Li

TL;DR: A novel Bayesian approach that solves the FL unlabel problem of data in clients

Abstract: Semi-supervised Federated Learning (SSFL) is a promising approach that allows clients to collaboratively train a global model in the absence of their local data labels. The key step of SSFL is the re-labeling where each client adopts two types of available models, namely global and local models, to re-label the local data. While various technologies such as using the global model or the average of two models have been proposed to conduct the re-labeling step, little literature delves deeply into the performance dominance and limitations of the two models. In this paper, we first theoretically and empirically demonstrate that the local model achieves higher re-labeling accuracy over local data while the global model can progressively improve the re-labeling performance by introducing the extra data knowledge of other clients. Based on these findings, we propose BSemiFL which re-labels the local data through the collaboration between the local and global model in a Bayesian approach. Specifically, to re-label any given local sample, BSemiFL first uses Bayesian inference to assess the closeness of the local/global model to the sample. Then, it applies a weighted combination of their pseudo labels, using the closeness as the weights. Theoretical analysis shows that the labeling error of our method is smaller than that of simply using the global model, the local model, or their simple average. Experimental results show that BSemiFL improves the performance by up to $9.8\%$ as compared to state-of-the-art methods.

Lay Summary:

Semi-supervised federated learning is a method that allows multiple users (we call them "clients") to collaborate in training an AI model. The advantage of this approach is that even if the data on each client's device is unlabeled, and without sharing the actual data, all clients can still jointly train a reasonably effective global model.
A key step in this process is called "re-labeling". In this step, each client uses two pre-trained models —— a global model shared by all users and a local model trained specifically on their own data —— to automatically assign labels to their unlabeled data. These newly labeled data are then used to further improve the model.
Past research has tried various ways to perform this labeling process —— for example, using only the global model, or taking an average of predictions from both the global and local models. However, few studies have deeply explored the strengths and limitations of these two models.
In our work, through theoretical analysis and experimental validation, we observed an important phenomenon:
The local model is better suited for labeling its own local data and achieves higher accuracy.
The global model, although initially less accurate, gradually improves its labeling performance as it incorporates knowledge from other clients' data.
Based on this finding, we propose a new method called BSemiFL. Its core idea is that instead of simply choosing one model or averaging their outputs, we use a technique called Bayesian inference to assess which model — the local or the global — is more "similar" to the current data sample. Based on this similarity, we compute weights for the two models and combine their predictions to generate more accurate pseudo-labels.
Theoretically, we show that this method results in fewer labeling errors compared to using only the local model, only the global model, or simply averaging the two. Experimental results also demonstrate that, compared to the current state-of-the-art methods, our approach improves performance by up to 9.8%.

Primary Area: Optimization->Large Scale, Parallel and Distributed

Keywords: Semi-supervised Federated Learning; Bayesian Approach

Submission Number: 5599