Finding Generalization Measures by Contrasting Signal and Noise

Jiaye Teng, Bohang Zhang, Ruichen Li, Yang Yuan

Keywords: generalization measure, signal and noise

TL;DR: A new generalization measure

Abstract: Generalization is one of the most fundamental challenges in deep learning, aiming to predict model performances on unseen data. Empirically, such predictions usually rely on a validation set, while recent works showed that an unlabeled validation set also works. Without validation sets, it is extremely difficult to obtain non-vacuous generalization bounds, which leads to a weaker task of finding generalization measures that monotonically relate to generalization error. In this paper, we propose a new generalization measure REF Complexity (RElative Fitting velocity between signal and noise), motivated by the intuition that a given model-algorithm pair may generalize well if it fits signal (e.g., true labels) fast while fitting noise (e.g., random labels) slow. Empirically, REF Complexity monotonically relates to test accuracy in real-world datasets without accessing additional validation sets, and achieves $-0.988$ correlation on CIFAR-10 and $-0.960$ correlation on CIFAR-100. We further theoretically verify the utility of REF Complexity under the regime of convex training with stochastic gradient descent.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Theory (eg, control theory, learning theory, algorithmic game theory)