Federated learning has become a popular machine learning paradigm with many potential real-life applications, including recommendation systems, the Internet of Things (IoT), healthcare, and self-driving cars. Though most current applications focus on classification-based tasks, learning personalized generative models remains largely unexplored, and their benefits in the heterogeneous setting still need to be better understood. 
This work proposes a novel architecture combining global client-agnostic and local client-specific generative models. We show that using standard techniques for training federated models, our proposed model achieves privacy and personalization that is achieved by implicitly disentangling the globally-consistent representation (i.e. content) from the client-dependent variations (i.e. style). Using such decomposition, personalized models can generate locally unseen labels while preserving the given style of the client and can predict the labels for all clients with high accuracy by training a simple linear classifier on the global content features.
Furthermore, disentanglement enables other essential applications, such as data anonymization, by sharing only content.
Extensive experimental evaluation corroborates our findings, and we also provide partial theoretical justifications for the proposed approach.

Partial Disentanglement with Partially-Federated GANs (PaDPaF)

Federated learning provides the ability to learn over heterogeneous user data in a distributed manner while preserving user privacy. However, its current client selection technique is a source of bias as it discriminates against slow clients. For starters, it selects clients that satisfy certain network and system-specific criteria, thus not selecting slow clients. Even when such clients are included in the training process, they either struggle with the training or are dropped altogether for being too slow. Our proposed idea looks to find a sweet spot between fast convergence and heterogeneity by looking at smart client selection and scheduling techniques.

FedSS: Federated Learning with Smart Selection of Clients

Continual Federated Learning (CFL) has garnered significant attention in recent years due to its potential in real-world scenarios where multiple clients (e.g., mobile phones, autonomous vehicles) continuously observe data in a dynamic environment. However, CFL suffers from catastrophic forgetting, where the model forgets previously learned knowledge in favor of new data. To address this issue, we propose a novel method called buffer-based Gradient Projection (FedGP) that mitigates catastrophic forgetting by replaying local buffer samples and using aggregated buffer gradients to preserve the previously learned knowledge across clients. Our method can be combined with a variety of existing continual learning methods, and boost their performance in the CFL setup. Our approach is evaluated on both standard benchmark datasets and a realistic streaming decentralized automotive dataset generated using CARLA and OpenFL. 

FedGP: Buffer-based Gradient Projection for Continual Federated Learning

We present VFL-PBM, a communication-efficient Vertical Federated Learning algorithm with Differential Privacy guarantees. VFL-PBM combines Secure Multi-Party Computation with the recently introduced Poisson Binomial Mechanism to protect parties' private datasets during model training. We analyze the end-to-end privacy and convergence behavior of our algorithm, and we provide the first theoretical characterization of the relationship between privacy, convergence error, and communication cost in differentially-private VFL. Our experiments show the VFL model performs well, with negligible decline in accuracy as we increase the privacy parameters.

Privacy Tradeoffs in Vertical Federated Learning

The majority of work in privacy-preserving federated learning (FL) has been focusing on horizontally partitioned datasets where clients share the same sets of features and can train complete models independently. However, in many interesting problems, such as financial fraud detection and disease detection, individual data points are scattered across different clients/organizations in vertical federated learning. Solutions for this type of FL require the exchange of gradients between participants and rarely consider privacy and security concerns, posing a potential risk of privacy leakage.  In this work, we present a novel design for training vertical FL securely and efficiently using state-of-the-art security modules for secure aggregation. We demonstrate empirically that our method does not impact training performance whilst obtaining \num{9.1e2} $\sim$ \num{3.8e4} speedup compared to homomorphic encryption (HE).

Efficient Vertical Federated Learning with Secure Aggregation

Machine learning models suffer from performance degradation when out-of-distribution (OOD) data samples, which do not come from their training data distributions, emerge after model deployment. A common practice called continual model refinement (CMR) in machine learning operations (MLOps) can alleviate such performance degradation by continuously refining deployed models over OOD data samples. However, few existing works on CMR tasks have considered federated learning (FL) settings where the OOD data samples are ubiquitous. To support CMR tasks in federated learning scenarios, we present a library called FedCMR, which includes a holistic pipeline that enables end-to-end CMR task evaluation ranging from data selection and labeling to model refinement and evaluation. We further show a case of integrating FedCMR with a federated learning ecosystem backed by the FedML production system (He et al., 2020). We hope that FedCMR could provide an efficient means for developing and evaluating federated CMR algorithms. We will open-source our library upon publication.


FedCMR: A Library for Federated Continual Model Refinement

Federated learning is a promising paradigm that allows multiple clients to collaboratively train a model without sharing the local data. However, the presence of heterogeneous devices in federated learning, such as mobile phones and IoT devices with varying memory capabilities, would limit the scale and hence the performance of the model could be trained. The mainstream approaches to address memory limitations focus on width-slimming techniques, where different clients train subnetworks with reduced widths locally and then the server aggregates the subnetworks. The global model produced from these methods suffers from performance degradation due to the negative impact of the actions taken to handle the varying subnetwork widths in the aggregation phase. In this paper, we introduce a memory-adaptive depth-wise learning solution in FL called FeDepth, which adaptively decomposes the full model into blocks according to the memory budgets of each client and trains blocks sequentially to obtain a full inference model. Our method outperforms state-of-the-art approaches, achieving 5% and more than 10% improvements in top-1 accuracy on CIFAR-10 and CIFAR-100, respectively. We also demonstrate the effectiveness of depth-wise fine-tuning on ViT. Our findings highlight the importance of memory-aware techniques for federated learning with heterogeneous devices and the success of depth-wise training strategy in improving the global model's performance.

Memory-adaptive Depth-wise Heterogenous Federated Learning

In conventional federated hyperdimensional computing (HDC), training larger models usually results in higher predictive performance but also requires more computational, communication, and energy resources. If the system resources are limited, one may have to sacrifice the predictive performance by reducing the size of the HDC model. The proposed resource-efficient federated hyperdimensional computing (RE-FHDC) framework alleviates such constraints by training multiple smaller independent HDC sub-models and refining the concatenated HDC model using the proposed dropout-inspired procedure. Our numerical comparison demonstrates that the proposed framework achieves a comparable or higher predictive performance while consuming less computational and wireless resources than the baseline federated HDC implementation.

Resource-Efficient Federated Hyperdimensional Computing

Federated learning (FL) has been proposed to enhance performance of local machine learning models across multiple devices while maintaining data privacy. One of the main challenges in FL is data heterogeneity, which limits the benefits of knowledge exchange among federated models. Data heterogeneity is a two-fold problem: Non-IID (not independent and identically distributed) data, and heterogeneous feature space. While personalized federated learning (PFL) solutions address challenges with non-IID data, most of the solutions require the same feature space. The few PFL solutions that are applicable in feature heterogeneity scenarios map all local domains into a new common feature space, which may result in degraded performance because of the negative transfer effects from unrelated local models based on a very different feature space. To address this limitation, we propose a novel clustered federated learning based on a Siamese graph convolutional neural network (FedSGCNN). We predict positive transfer between clients using an SGCNN in order to create a distance matrix for clustering. This network-based prediction is more accurate as compared to other distance measures which fail to capture the structure of models. When there is feature space heterogeneity, we show that FedSGCNN outperforms the latest work by 1.2% in the Boston Housing dataset and 2.8% in the Obesity Level dataset.


Clustered Federated Learning for Heterogeneous Feature Spaces using Siamese Graph Convolutional Neural Network Distance Prediction

Federated learning (FL) is a popular distributed deep learning framework which enables personalized experiences across a wide range of web clients and mobile/IoT devices. However, FL-based methods are challenged by the compute resources on client devices given the exploding growth in size of state-of-the-art models (eg. billion parameter models). Split Learning (SL), a recent framework, reduces client compute load by splitting model training between client and server. This flexibility is useful for low-compute setups but is achieved at the cost of massive increase in bandwidth consumption. This split also results in sub-optimal performance, especially when data across clients is heterogeneous. The goal of this paper is to make SL a viable alternative to FL. Specifically, we introduce adaptive split learning (AdaSplit) which enables efficiently scaling SL to low-resource scenarios by reducing bandwidth consumption and improving performance across heterogenous clients. We validate the effectiveness of AdaSplit under limited resources through extensive experimental comparison with strong federated and split learning baselines. Finally, we also present sensitivity analyses of key design choices in AdaSplit which highlight the ability of AdaSplit to adapt to variable resource budgets.

MLSys 2023 Workshop FLSys Submissions