Generalization of Gibbs and Langevin Monte Carlo Algorithms in the Interpolation Regime

ICLR 2026 Conference Submission17910 Authors

Keywords: Generalization bounds, PAC-Bayesian analysis, Gibbs posterior, Langevin Monte Carlo, stochastic gradient Langevin dynamics, interpolation regime

TL;DR: We give data-dependent generalization bound for the Gibbs algorithm in the interpolation regime, show their stability under Langevin Monte Carlo, and validate them on MNIST and CIFAR-10.

Abstract: The paper provides data-dependent bounds on the test error of the Gibbs algorithm in the overparameterized interpolation regime, where low training errors are also obtained for impossible data, such as random labels in classification. The bounds are stable under approximation with Langevin Monte Carlo algorithms. Experiments on the MNIST and CIFAR-10 datasets verify that the bounds yield nontrivial predictions on true labeled data and correctly upper bound the test error for random labels. Our method indicates that generalization in the low-temperature, interpolation regime is already signaled by small training errors in the more classical high temperature regime.

Primary Area: learning theory

Submission Number: 17910