Stability, Complexity and Data-Dependent Worst-Case Generalization Bounds

Mario Tuci, Benjamin Dupuis, Lennart Bastian, Nassir Navab, Tolga Birdal, Umut Simsekli

Keywords: learning theory, topological generalization, algorithmic stability

TL;DR: We introduce a novel framework for deriving data- and algorithm-dependent worst-case generalization bounds. As a result, our approach yields fully computable topological generalization bounds.

Abstract:

Providing generalization guarantees for stochastic optimization algorithms remains a key challenge in learning theory.
Recently, numerous works demonstrated the impact of the geometric properties of optimization trajectories on generalization performance.
These works propose worst-case generalization bounds in terms of various notions of intrinsic dimension and/or topological complexity, which were found to empirically correlate with the generalization error.
However, most of these approaches involve intractable mutual information terms, which limit a full understanding of the bounds.
In contrast, some authors built on algorithmic stability to obtain worst-case bounds involving geometric quantities of a combinatorial nature, which are impractical to compute.
In this paper, we address these limitations by combining empirically relevant complexity measures with a framework that avoids intractable quantities.
To this end, we introduce the concept of \emph{random set stability}, tailored for the data-dependent random sets produced by stochastic optimization algorithms.
Within this framework, we show that the worst-case generalization error can be bounded in terms of (i) the random set stability parameter and (ii) an empirically relevant, data- and algorithm-dependent complexity measure of the corresponding random set.
Moreover, our framework improves existing topological generalization bounds by recovering previous complexity notions without relying on mutual information terms.
Through a series of experiments in practically relevant settings, we validate our theory by evaluating the tightness and of our bounds and the interplay between topological complexity and stability.

Primary Area: learning theory

Submission Number: 2679