Towards Robust Influence Functions with Flat Validation Minima

Xichen Ye, Yifan Wu, WEIZHONG ZHANG, Cheng Jin, Yifan Chen

Abstract:

The Influence Function (IF) is a widely used technique for assessing the impact of individual training samples on model predictions.
However, existing IF methods often fail to provide reliable influence estimates in deep neural networks, particularly when applied to noisy training data.
This issue does not stem from inaccuracies in parameter change estimation, which has been the primary focus of prior research, but rather from deficiencies in loss change estimation, specifically due to the sharpness of validation risk.
In this work, we establish a theoretical connection between influence estimation error, validation set risk, and its sharpness, underscoring the importance of flat validation minima for accurate influence estimation.
Furthermore, we introduce a novel estimation form of Influence Function specifically designed for flat validation minima.
Experimental results across various tasks validate the superiority of our approach.

Lay Summary:

Modern machine learning systems are trained on massive datasets, but not all data points are good: some are noisy, mislabeled, or even harmful, reducing model quality. Identifying which training examples help or hurt has become a major challenge, especially as models grow more complex.

We tackled this problem using a technique called the “influence function,” which measures how much each training example affects a model’s predictions. However, we found that existing influence function methods often fail on modern deep networks because they assume overly simple conditions. To solve this, we proposed a new influence function formulation that specifically accounts for the ''flatness validation minima'', making influence estimates more accurate and robust.

Our method improves the ability to detect harmful or misleading data points, enabling researchers to clean datasets, debug models, and improve fairness. Importantly, it scales well to large models and diverse tasks, including image generation and language processing. By making model training more trustworthy and interpretable, our work can help build AI systems that are safer and more reliable for society.

Link To Code: https://github.com/Virusdol/IF-FVM

Primary Area: Deep Learning->Robustness

Keywords: Influence Function

Submission Number: 3353