A Closer Look at NTK Alignment: Linking Phase Transitions in Deep Image Regression

Giuseppe Castiglione, Christopher Buckley, Ivor J A Simpson

Keywords: neural tangent kernel, phase transitions, learning dynamics

TL;DR: We mathematically model the evolution of the NTK in a fully deep neural network trained to represent natural images.

Abstract: Deep neural networks trained with gradient descent exhibit varying rates of learning for different patterns. However, the complexity of fitting models to data makes direct elucidation of the dynamics of learned patterns challenging. To circumvent this, many works have opted to characterize phases of learning through summary statistics known as order parameters. In this work, we propose a unifying framework for constructing order parameters based on the Neural Tangent Kernel (NTK), in which the relationship with the data set is more transparent. In particular, we derive a local approximation of the NTK for a class of deep regression models (SIRENs) trained to reconstruct natural images. In so doing, we analytically connect three seemingly distinct phase transitions: the emergence of wave patterns in residuals (a novel observation), loss rate collapse, and NTK alignment. Our results provide a dynamical perspective on the observed biases of SIRENs, and deep image regression models more generally.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 7165