Provable Data-driven Hyperparameter Tuning for Deep Neural Networks
Keywords: learning theory, data-driven algorithm design, hyperparameter tuning, neural architecture search, graph neural networks, sample complexity
Abstract: Modern machine learning algorithms, especially deep learning-based techniques, typically involve careful
hyperparameter tuning to achieve the best performance. Despite the surge of intense interest in practical
techniques like Bayesian optimization and random search-based approaches to automating this laborious and
compute-intensive task, the fundamental learning-theoretic complexity of tuning hyperparameters for deep
neural networks is poorly understood. Inspired by this glaring gap, we initiate the formal study of hyperparameter tuning complexity in deep learning through a recently introduced lens of data-driven algorithm design. We assume that we have a series of deep learning tasks, and we have to tune hyperparameters to do well on average over the distribution of
tasks. A major difficulty is that the loss as a function of the hyperparameter is very volatile and furthermore,
it is given implicitly by an optimization problem over the model parameters. This is unlike previous work
in data-driven design, where one can typically explicitly model the algorithmic behavior as a function of
the hyperparameters. To tackle this we introduce a new technique to characterize the discontinuities and
oscillations of the loss function on any fixed problem instance as we vary the hyperparameter; our analysis
relies on subtle concepts including tools from differential geometry and constrained optimization. This can be
used to show that the intrinsic complexity of the corresponding family of loss functions is bounded. We instantiate
our results and provide the first precise sample complexity bounds for concrete applications—tuning a hyperparameter that interpolates neural activation functions and setting the kernel
parameter in graph neural networks.
Primary Area: learning theory
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12854
Loading