A Non-vacuous Test Error Guarantee for Deep Learning without Altering the Model

Khoat Than, Dat Phan

Keywords: Deep Learning, generalization bounds, Error bounds, Model-dependent bounds, non-vacuous

Abstract: Deep neural networks (NN) with millions or billions of parameters can perform really well on unseen data, after being trained from a finite training set. Various prior theories have been developed to explain such excellent ability of NNs, but do not provide a meaningful bound on the test error. Some recent theories are non-vacuous under some stringent assumptions and extensive modification (e.g. compression, quantization) to the trained model of interest. Therefore, those prior theories provide a guarantee for the modified models only. In this paper, we present two novel bounds on the true error of a model. One of our bounds can be exactly computable from the training set only, without altering the model, and hence provides a theoretical guarantee for a trained model. Our approach is to decompose the data space into different local areas to approximate the local errors by using training samples in a controlled way, then use those local errors to approximate the true error of a model. Our bounds are verified on 32 modern NNs, which were trained by Pytorch on the ImageNet dataset. The exactly computable bound is found to be non-vacuous. To the best of our knowledge, this is the first non-vacuous bound at this large scale (NNs with more than 600M parameters, ImageNet), without altering those 32 trained models.

Primary Area: learning theory

Submission Number: 24160