Using Objective Bayesian Methods to Determine the Optimal Degree of Curvature within the Loss Landscape

Devon Jarvis, Richard Klein, Benjamin Rosman

Keywords: Objective Bayes, Information Geometry, Artificial Neural Networks

TL;DR: We reflect that the widest point in the parameter landscape corresponds to a model which has overfit the training data, we propose a new determinant of the optimal width within the parameter landscape.

Abstract: The efficacy of the width of the basin of attraction surrounding a minimum in parameter space as an indicator for the generalizability of a model parametrization is a point of contention surrounding the training of artificial neural networks, with the dominant view being that wider areas in the landscape reflect better generalizability by the trained model. In this work, however, we aim to show that this is only true for a noiseless system and in general the trend of the model towards wide areas in the landscape reflect the propensity of the model to overfit the training data. Utilizing the objective Bayesian (Jeffreys) prior we instead propose a different determinant of the optimal width within the parameter landscape determined solely by the curvature of the landscape. In doing so we utilize the decomposition of the landscape into the dimensions of principal curvature and find the first principal curvature dimension of the parameter space to be independent of noise within the training data.