Bilevel Optimization for Hyperparameter Learning in Supporting Vector Machines

Lei Huang, Jiawang Nie, Jiajia Wang, Suhan Zhong

Track: Full Paper (8 pages)

Keywords: Bilevel optimization, Hyperparameter Learning, Supporting Vector Machine, Lagrange Multiplier Expressions

TL;DR: We propose a novel single-level reformulation for bilevel problems with convex lower-level objectives and linear constraints, which eliminates auxiliary variables and preserves the original problem dimension.

Abstract: Bilevel optimization is central to many machine learning tasks, including hyperparameter learning and adversarial training. We present a novel single-level reformulation for bilevel problems with convex lower-level objective functions and linear constraints. Our method eliminates auxiliary Lagrange multiplier variables by expressing them in terms of the original decision variables, which allows the reformulated problem to preserve the same dimension as the original problem. We applied our method to support vector machines (SVMs) and evaluated it on several benchmark tasks, demonstrating efficiency and scalability.

Submission Number: 3