Inconsistency-Aware Minimization: Improving Generalization with Unlabeled Data

ICLR 2026 Conference Submission17047 Authors

Keywords: Generalization, Regularization, Training Method, Deep Learning, Unlabeled Data

Abstract:

Estimating the generalization gap and developing optimization methods that improve generalization are crucial for deep learning models, for both theoretical understanding and practical applications.
Leveraging unlabeled data for these purposes offers significant advantages in real-world scenarios.
This paper introduces a novel generalization measure, $\textit{local inconsistency}$, derived from an information-geometric perspective on the parameter space of the neural networks. A key feature of local inconsistency is that it can be computed without explicit labels.
We establish theoretical underpinnings by connecting local inconsistency to Fisher information matrix and loss Hessian.
Empirically, we demonstrate that local inconsistency correlates with the generalization gap.
Based on these findings, we propose Inconsistency-Aware Minimization (IAM), which incorporates local inconsistency into the training objective.
We demonstrate that in standard supervised learning settings, IAM enhances generalization, achieving performance comparable to that of existing methods such as Sharpness-Aware Minimization.
Furthermore, IAM exhibits efficacy in semi- and self-supervised learning scenarios, where the local inconsistency is computed from unlabeled data.

Primary Area: optimization

Submission Number: 17047