Personalized Federated Learning of Probabilistic Models: A PAC-Bayesian Approach

Mahrokh Ghoddousi Boroujeni, Andreas Krause, Giancarlo Ferrari-Trecate

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Federated Learning, Probabilistic Modeling, PAC-Bayesian Learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A personalized federated learning algorithm for learning probabilistic models within a PAC-Bayesian framework that utilizes differential privacy to handle data-dependent priors.

Abstract:

Federated learning aims to infer a shared model from private and decentralized data stored locally by multiple clients.
Personalized federated learning (PFL) goes one step further by adapting the global model to each client's data, enhancing the model's fit for different clients.
A significant level of personalization is required for highly heterogeneous clients, but can be challenging to achieve especially when they have small datasets.
To address this problem, we propose a PFL algorithm named *PAC-PFL* for learning probabilistic models within a PAC-Bayesian framework that utilizes differential privacy to handle data-dependent priors.
Our algorithm collaboratively learns a shared hyper-posterior and regards each client's posterior inference as the personalization step.
By establishing and minimizing a generalization bound on the average true risk of clients, PAC-PFL effectively combats over-fitting.
Empirically, PAC-PFL achieves accurate and well-calibrated predictions as demonstrated through experiments on a highly heterogeneous dataset of photovoltaic panel power generation and the FEMNIST dataset.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4010