Training-Free Determination of Network Width via Neural Tangent Kernel

Tatsumi Sunada, Toshihiko Yamasaki, Atsuto Maki

Keywords: neural tangent kernel, kernel regression, smallest eigenvalue, generalization error

Abstract: Determining an appropriate size for an artificial neural network under computational constraints is a fundamental challenge. This paper introduces a practical metric, derived from Neural Tangent Kernel (NTK), for estimating the minimum necessary network width with respect to test loss *prior to training*. We provide both theoretical and empirical evidence that the smallest eigenvalue of the NTK strongly influences test loss in wide but finite-width neural networks. Based on this observation, we define an NTK-based metric computed at initialization to identify what we call *cardinal width*, i.e., the width of a network at which generalization performance saturates. Our experiments across multiple datasets and architectures demonstrate the effectiveness of this metric in estimating the *cardinal width*.

Primary Area: learning theory

Submission Number: 6449