Characterizing trainability, expressivity, and generalization of neural architecture with metrics from neural tangent kernel

Xinyu Chen, Jinlong Li, Faiza Amin, Huanhuan Chen, Qiuju Chen

Keywords: neural tangential kernel, neural architecture search, trainability, expressivity, generalization

TL;DR: We introduce a training-free metric derived from neural tangential kernel to characterize trainability, expressivity, and generalization of neural architecture.

Abstract:

Zero-shot neural architecture search aims to predict multiple characteristics of neural architectures using proxy indicators without actual training, yet most methods focus on evaluating only a single characteristic of neural networks.
Since the Neural Tangent Kernel (NTK) offers a promising theoretical framework for understanding the characteristics of neural networks, we propose NTK-score, including three metrics derived from NTK's eigenvalues and kernel regression, to assess three critical characteristics: trainability, expressivity, and generalization.
Moreover, to exploit three metrics of our NTK-score, we employ the Borda Count approach on our NTK-score to rank architectures in neural architecture search.
Compared with state-of-the-art proxies, experimental results demonstrate that the NTK-score correlates well with both the accuracy and training time of architectures, and exhibits excellent performance across various search spaces and methods, including NAS-bench-201, DARTS, and ResNet, as well as pruning, reinforce, and evolutionary algorithm.

Primary Area: learning theory

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10989