Lower-level Duality Based Penalty Methods for Bilevel Hyperparameter Optimization

Haochen Xu, Rujun Jiang, Chenhao Yang

Keywords: Bilevel Optimization, Nonsmooth Optimization, Nonconvex Optimization, Hyperparameter Optimization

TL;DR: We propose first-order single-loop algorithms based on penalization for bilevel hyperparameter optimization.

Abstract: Hyperparameter optimization (HO) is a critical task in machine learning and can be formulated as a bilevel optimization problem. However, many existing algorithms for addressing nonsmooth lower-level problems involve solving sequential subproblems, which are computationally expensive. To address this challenge, we propose penalty methods for solving HO, leveraging strong duality between the lower-level problem and its dual. We show that the penalized problem closely approximates the optimal solutions of the original HO under certain conditions. Moreover, we develop first-order single-loop algorithms to solve the penalized problems efficiently. Theoretically, we establish the convergence of the proposed algorithms. Numerical experiments demonstrate the efficiency and superiority of our method.

Primary Area: Optimization (e.g., convex and non-convex, stochastic, robust)

Submission Number: 7332