Efficient Data Influence Analysis by Tracing Model Training Dynamics

ICLR 2026 Conference Submission21623 Authors

Keywords: influence analysis, training dynamic, data attribution, efficiency

TL;DR: In this paper, we propose a novel, simple, and efficient approach to influence analysis that learns influence representations using contrastive learning from the model's training dynamics.

Abstract: Quantifying the impact of training data is essential for understanding model behavior and optimizing the training process. Despite extensive research into influence estimation, existing methods often rely on repeated training or gradient analysis, which results in prohibitive computational and memory costs and limits their applicability to large-scale models and datasets. In this paper, we explore a new perspective on influence estimation by distilling influence signals from training dynamics, i.e., the model’s predictions on individual examples throughout training. We propose an influence estimation approach that uses contrastive learning to project the observed influence into a representation space, where the proximity between data points reflects their influence strength. Our approach is simple, efficient, and scalable, requiring neither gradient computation nor assumptions about the optimizers. We validate our approach across various tasks and datasets, demonstrating its ability to estimate influence effectively, scalability to large models, and utility in downstream applications, such as mislabeled data identification, influential data selection, and data attribution.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 21623