Rescaled Influence Functions: Accurate Data Attribution in High Dimension

Ittai Rubinstein, Samuel B. Hopkins

Keywords: data attribution, influence functions, high-dimensional models, overparameterized models, model interpretability, robustness, training data influence, rescaled influence functions, data poisoning detection, machine learning theory

TL;DR: We evaluate Rescaled Influence Functions (RIF), a fast and accurate alternative to traditional influence functions for data attribution, particularly effective in high-dimensional settings where standard influence methods fail.

Abstract:

How does the training data affect a model's behavior?
This is the question we seek to answer with *data attribution*.
The leading practical approaches to data attribution are based on *influence functions* (IF).
IFs utilize a first-order Taylor approximation to efficiently predict the effect of removing a set of samples from the training set without retraining the model, and are used in a wide variety of machine learning applications.
However, especially in the high-dimensional regime (# params $\geq \Omega($# samples$)$), they are often imprecise and tend to underestimate the effect of sample removals, even for simple models such as logistic regression.
We present *rescaled influence functions* (RIF) -- a tool for data attribution which can be used as a drop-in replacement for influence functions, with little computational overhead but significant improvement in accuracy.
We compare IF and RIF on a range of real-world datasets, showing that RIFs offer significantly better predictions in practice, and present a theoretical analysis explaining this improvement.
Finally, we present a simple class of data poisoning attacks that would fool IF-based detections but would be detected by RIF.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 13325