One Measure, Many Bounds: Bridging TV, Variance, and Mutual Information

ICLR 2026 Conference Submission24118 Authors

Keywords: Generalization Bounds, Information Theory, Algorithmic Stability, Statistical Learning Theory, Variance Analysis

TL;DR: We unify generalization bounds based on total variation, mutual information, and variance under a single tunable measure, providing a new information-theoretic perspective on algorithmic stability.

Abstract: Understanding the generalization of machine learning algorithms remains a fundamental challenge. While mutual information provides a powerful lens for analysis, we introduce a more flexible, one-parameter family of information-theoretic generalization bounds based on the vector-valued $L_p$-norm correlation measure, $V_\alpha$. Our framework unifies and interpolates between several existing information-theoretic guarantees, including those based on total variation and Rényi information. The primary conceptual contribution of our work emerges at $\alpha=2$, where our framework yields a novel and intuitive variance-based bound. This result establishes the variance of the algorithm's output distribution, $\mathrm{Var}_S[p(w|S)]$, as a direct, data-dependent measure of algorithmic stability. We prove that this measure directly controls the generalization error, thus providing a new, information-theoretic perspective on how unstable (high-variance) algorithms fail to generalize. Extensive simulations demonstrate that our bounds, particularly for $\alpha=2$, can be significantly tighter than classical mutual information guarantees.

Primary Area: learning theory

Submission Number: 24118