Nonconvex Meta-optimization for Deep Learning

Published: 16 Jun 2024, Last Modified: 23 Jul 2024HiLD at ICML 2024 PosterEveryoneRevisionsBibTeXCC BY 4.0
Keywords: deep learning, online learning, online control
Abstract: Hyperparameter tuning in mathematical optimization is a notoriously difficult problem. Recent tools from online control give rise to a provable methodology for hyperparameter tuning in convex optimization called meta-optimization. In this work, we extend this methodology to nonconvex optimization and the training of deep neural networks. We present an algorithm for nonconvex meta-optimization that leverages the reduction from nonconvex optimization to convex optimization, and investigate its applicability for deep learning tasks on academic-scale datasets.
Student Paper: Yes
Submission Number: 42
Loading