The Effect of the Intrinsic Dimension on the Generalization of Quadratic ClassifiersDownload PDF

21 May 2021, 20:49 (modified: 26 Oct 2021, 10:40)NeurIPS 2021 PosterReaders: Everyone
Keywords: intrinsic dimension, generalization, rademacher complexity, statistical learning theory
TL;DR: We show that the complexity of nuclear-norm regularized quadratic functions adapts to the intrinsic dimension of the data distribution.
Abstract: It has been recently observed that neural networks, unlike kernel methods, enjoy a reduced sample complexity when the distribution is isotropic (i.e., when the covariance matrix is the identity). We find that this sensitivity to the data distribution is not exclusive to neural networks, and the same phenomenon can be observed on the class of quadratic classifiers (i.e., the sign of a quadratic polynomial) with a nuclear-norm constraint. We demonstrate this by deriving an upper bound on the Rademacher Complexity that depends on two key quantities: (i) the intrinsic dimension, which is a measure of isotropy, and (ii) the largest eigenvalue of the second moment (covariance) matrix of the distribution. Our result improves the dependence on the dimension over the best previously known bound and precisely quantifies the relation between the sample complexity and the level of isotropy of the distribution.
Supplementary Material: pdf
Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
Code: zip
7 Replies