The Eigenlearning Framework: A Conservation Law Perspective on Kernel Ridge Regression and Wide Neural Networks

James B Simon, Madeline Dickens, Dhruva Karkada, Michael Deweese

Keywords: kernel regression, kernel ridge regression, wide neural networks, neural tangent kernel, ntk, generalization, conservation laws, learnability

TL;DR: We identify a conserved quantity in kernel ridge regression and leverage it to develop a theory of generalization, concluding with various applications of our framework.

Abstract:

We derive simple closed-form estimates for the test risk and other generalization metrics of kernel ridge regression (KRR). Relative to prior work, our derivations are greatly simplified and our final expressions are far more interpretable. These improvements are enabled by our identification of a sharp conservation law which limits the ability of KRR to learn any orthonormal basis of functions. Test risk and other objects of interest are expressed in a transparent, interpretable way in terms of our conserved quantity evaluated in the kernel eigenbasis.
We use our improved framework to:
i) provide a theoretical explanation for the ``deep bootstrap" of Nakkiran et al (2020),
ii) prove a new result regarding the hardness of the classic parity problem,
iii) fashion a theoretical tool for the study of adversarial robustness, and
iv) draw a tight analogy between KRR and a well-studied system in statistical physics.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning