Stochastic Hyperparameter Optimization through Hypernetworks

Jonathan Lorraine, David Duvenaud

Feb 12, 2018 (modified: Feb 15, 2018) ICLR 2018 Workshop Submission readers: everyone Show Bibtex
  • Abstract: Machine learning models are often tuned by nesting optimization of model weights inside the optimization of hyperparameters. We give a method to collapse this nested optimization into joint stochastic optimization of weights and hyperparameters. Our process trains a neural network to output approximately optimal weights as a function of hyperparameters. We show that our technique converges to locally optimal weights and hyperparameters for sufficiently large hypernets. We compare this method to standard hyperparameter optimization strategies.
  • TL;DR: We train a neural network to output approximately optimal weights as a function of hyperparameters.
  • Keywords: hypernetworks, hyperparameter optimization, metalearning, neural networks, Bayesian optimization, game theory, optimization
0 Replies

Loading