MeFBO: A Moreau Envelope Based First-Order Stochastic Gradient Method for Nonconvex Federated Bilevel Optimization
Keywords: Federated Bilevel Optimization, Nonconvex, Hessian-free, Convergence Analysis, Linear Speedup
Abstract: Federated Bilevel Optimization (FBO) enables training machine learning models with nested structures across distributed devices while preserving data privacy. However, current FBO methods often impose restrictive assumptions, particularly the requirement of strong convexity in the lower-level objective. To overcome this limitation, we propose a first-order stochastic gradient method for general FBO problems, leveraging a Moreau envelope-based min-max optimization reformulation to handle potentially non-convex lower-level objectives. Unlike implicit gradient methods, our approach eliminates the need for second-order derivative information. We also establish rigorous theoretical guarantees for convergence rate and communication complexity, demonstrating linear speedup as the number of devices increases. Numerical experiments validate the effectiveness and efficiency of our method, showing comparable or superior performances in challenging scenarios, including federated loss function tuning on imbalanced datasets and federated hyper-representation.
Supplementary Material: zip
Primary Area: optimization
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 8967
Loading