Learning Coefficients, Fractals, and Trees in Parameter Space

Max Hennick, Matthias Dellago

Keywords: Singular Learning Theory, fractals, trees, symbolic dynamics

TL;DR: We relate the learning coefficient to the box counting dimension, and then link this to infinite depth trees and cylinder sets.

Abstract: It is well known that the local geometric structure of the loss surface about a particular value of the parameter space for a deep learning model (and other singular models) determines many of the behavioural properties of the model with that parameter value. In recent years the learning coefficient has emerged as a particularly important geometric invariant for predicting model properties. In this work we explore the interpretation of the learning coefficient as a fractal dimension of the loss surface, and show how it relates to more classical notions of fractal dimensions like the box counting dimension. Using this we show that there is a natural correspondence between the low loss parameters and an infinite depth, locally finite tree. We then use this to reframe the learning coefficient in terms of cylinder sets, which draws links between the geometry of parameter space, information theory, symbolic dynamics, and probability on trees.

Confirmation: I confirm that I and my co-authors have read the policies are releasing our work under a CC-BY 4.0 license.

Submission Number: 2