Understanding Generalization via Leave-One-Out Conditional Mutual Information

Mahdi Haghifam, Shay Moran, Daniel M. Roy, Gintare Karolina Dziugaite

Abstract: We study the mutual information between the output of a learning algorithm and its n training data, conditional on a supersample of n+1 i.i.d. data from which the training data is chosen at random without replacement. We show that this variant of the conditional mutual information (CMI) of an algorithm (Steinke and Zakynthinou, 2020), which we dub leave-one-out CMI, determines the mean generalization error of any interpolating learning algorithm with a bounded loss function. For interpolating and non-interpolating classifiers, we demonstrate that bounded leave-one-out CMI implies generalization. Finally, as an application, we analyze the population risk of the One-Inclusion-Graph Algorithm, a transductive learning algorithm for VC classes in the realizable setting, and prove that our framework is the first information-theoretic framework that is able to achieve the optimal bound for learning VC classes in the realizable setting.