How Data Influence Changes in Training? Time-Varying Influence Measurement

Jie Xu, Zihan Wu, Wenbo Pan, Hua Wang, Cong Wang, Xiaohua Jia

Keywords: Time-varying Influence, Training Dynamics, Model Interpretability, Data Influence Analysis

TL;DR: Moving beyond traditional static, overall analysis, TIM precisely measures LOO data influence within arbitrary training windows, revealing how influence changes during training.

Abstract: Existing data influence analyses are static, measuring the global, cumulative influence of training data on fully trained models while leaving dynamic changes during training a black box. We propose Time-varying Influence Measurement (TIM), the first framework measuring how data influence changes during training. TIM operates on arbitrary local windows, estimating how removing a training point within a window affects model parameters, and then projects these parameter deviations onto task-relevant functional responses (e.g., test loss) via query vectors. We establish theoretical error bounds under non-convex and non-converged conditions. Experiments show that: 1) TIM estimates loss changes more accurately than prior methods and closely matches Leave-One-Out (LOO) retraining; 2) Data influence is time-varying, exhibiting different patterns including Early Influencers, Late Bloomers, Stable Influencers, and Highly Fluctuating patterns; 3) Global or longer windows are not necessarily better, as small-window TIM achieves better performance in corrupted data identification while reducing cost by 95\%.

Primary Area: interpretability and explainable AI

Submission Number: 3557