Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks

Florian A. Hölzl, Daniel Rueckert, Georgios Kaissis

Keywords: deep learning, generalization, classification

Abstract:

Robust validation metrics remain essential in contemporary deep learning, not only to detect overfitting and poor generalization, but also to monitor training dynamics.
In the supervised classification setting, we investigate whether interactions between training data and model weights can yield such a metric that both tracks generalization during training and attributes performance to individual training samples.
We introduce Gradient-Weight Alignment (GWA), quantifying the coherence between per-sample gradients and model weights.
We show that effective learning corresponds to coherent alignment, while misalignment indicates deteriorating generalization.
GWA is efficiently computable during training and reflects both sample-specific contributions and dataset-wide learning dynamics.
Extensive experiments show that GWA accurately predicts optimal early stopping, enables principled model comparisons, and identifies influential training samples, providing a validation-set-free approach for model analysis directly from the training data.

Primary Area: General machine learning (supervised, unsupervised, online, active, etc.)

Submission Number: 4085