Anarchic Federated Bilevel Optimization

Dongsheng Li, Xiaowen Gong, Shiwen Mao, Yang Zhou

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: asynchronous, bilevel optimization, heterogenous local iterations, partial participation

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Rapid federated bilevel optimization (FBO) developments have attracted much attention in various emerging machine learning and communication applications. Existing work on FBO often assumes that clients participate in the learning process with some particular pattern (such as balanced participation), and/or in a synchronous manner, and/or with homogeneous local iteration numbers, which might be
hard to hold in practice. This paper proposes a novel Asynchronous Federated Bilevel Optimization (AFBO) algorithm, which allows clients to 1) participate in any inner or outer rounds; 2) participate asynchronously; and 3) participate with any number of local iterations. The proposed AFBO algorithm enables clients to flexibly participate in FBO training. We provide a theoretic analysis of the learning loss of AFBO and the result shows that the AFBO algorithm can achieve a convergence rate of $\mathcal{O}(\sqrt{\frac1T})$, which matches that of the existing benchmarks. Numerical studies are conducted to verify the efficiency of the proposed algorithm.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2013