Differentiable Engine for Tri-level Optimization

Yuntian Gu, Xuzheng Chen, Junqing Chen

Keywords: Differentiable Engine for Tri-level Optimization

Abstract: Bi-level optimization, a concept extensively explored in economics, mathematics, and computer science, has recently gained renewed interest in machine learning. It shows promise in various machine learning applications, including hyperparameter tuning and continual learning. This article provides an overview of two principal forms of bi-level optimization, distinguished by whether the lower level optimization is parameterized by upper level parameters. We discuss the conventional gradient-based solutions and propose their extension to tri-level optimization, potentially applicable in multi-stage game scenarios. We conduct extensive experiments and show that our algorithm indeed outperforms vanilla alternative.

Submission Number: 232