On the Accuracy of Newton Step and Influence Function Data Attributions

Ittai Rubinstein, Samuel B. Hopkins

Keywords: data attribution, analysis, influence functions, newton step

TL;DR: We introduce the first analysis of NS and IF data attributions that does not scale poorly with the regularization coefficient.

Abstract:

Data attribution aims to explain model predictions by estimating how they would change if certain training points were removed, supporting a wide range of applications, from interpretability and credit assignment to unlearning and privacy.

Even in the relatively simple case of linear models with normally distributed features, existing analyses of leading data attribution methods such as Influence Functions (IF) and single Newton Step (NS) remain limited.
Current bounds require a strong dependence on the global strong convexity parameter $\lambda$, which is often very small in practice and scales like $O(d/n)$ for well-behaved regressions with dimension $d$ and $n$ samples.
These bounds also scale poorly with the number of removed samples $k$ and with $d$. Making the dependence on $\lambda$ explicit reveals that existing results are very loose:
$$ \text{Existing Bounds} = \Omega\left(\frac{k^2 d}{\lambda^3 n^2} \right) = \Omega\left(\frac{k^2 n}{d^2}\right).$$

We introduce new analytic tools for bounding the errors of NS, yielding substantially tighter results that do not depend on the global strong convexity $\lambda$.
Moreover, we show that for logistic regressions with normally distributed features, our bounds also scale much more favorably with $k$ and $d$:

$$\text{New Bound} = \tilde{O}\left( \frac{k}{n^2} \right).$$

We show that our bounds are tight up to poly-logarithmic factors, that they also yield similarly tight bounds on the accuracy of IF and provide the first theoretical explanation for the empirical observation that NS is more accurate than IF.

Primary Area: learning theory

Submission Number: 21695