Selective Prediction via Training Dynamics

Stephan Rabanser, Anvith Thudi, Kimia Hamidieh, Adam Dziedzic, Israfil Bahceci, Akram Bin Sediq, Hamza Sokun, Nicolas Papernot

Keywords: training dynamics, selective prediction, uncertainty, abstention, rejection

TL;DR: We achieve state-of-the-art selective prediction by rejecting inputs with unstable late-stage predictions, using training dynamics alone—no architectural changes, domain-specific tuning, or retraining required.

Abstract: Selective prediction aims to reject inputs a model is likely to misclassify, balancing input coverage (how many points are accepted) with utility (performance on accepted inputs). Existing methods often modify model architectures or objectives, limiting practical use and introducing unwanted interactions with existing losses. In contrast, we show that state-of-the-art performance can be achieved by analyzing a model’s discretized training dynamics. Our framework monitors the instability of intermediate checkpoint predictions relative to the final model and rejects inputs with excessive late-stage disagreement. The approach is domain-agnostic, requires no train-time changes, and can be combined with existing methods. Experiments across image, regression, and time series tasks show that our method outperforms prior state-of-the-art utility–coverage trade-offs.

Student Paper: Yes

Submission Number: 4