Guiding Neural Network Initialization via Marginal Likelihood Maximization

Anthony Tai, Chunfeng Huang

Keywords: Neural networks, Gaussian processes, model initialization, marginal likelihood

Abstract: We propose a simple, data-driven approach to help guide hyperparameter selection for neural network initialization. We leverage the relationship between neural network and Gaussian process models having corresponding activation and covariance functions to infer the hyperparameter values desirable for model initialization. Our experiment shows that marginal likelihood maximization provides recommendations that yield near-optimal prediction performance on MNIST classification task under experiment constraints. Furthermore, our empirical results indicate consistency in the proposed technique, suggesting that computation cost for the procedure could be significantly reduced with smaller training sets.

One-sentence Summary: We propose using Gaussian process marginal likelihood maximization to recommend hyperparameter values for initialization of the corresponding neural network.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=ijzWY1nA_9