Influence Estimation in Statistical Models Using the Fisher Information Matrix

Omri Lev, Ashia C. Wilson

Abstract: Quantifying how infinitesimal perturbations of training data affect a model is key to diagnosing and improving learning systems. This task was addressed via the notion of influence functions \citep{Hampel_Influence, KohLiang_Influence_DL, koh2019accuracy, Bae_PBRF_Influence}. Following classical works, whenever the underlying problem can be cast as a weighted empirical risk minimization problem, many such influence estimators rely on the Fisher Information Matrix (FIM). Following these lines, we provide a new accuracy analysis that characterizes the asymptotic behavior of FIM-based influence estimators and compare these to Hessian-based influence estimators, while we further extend the theory to objectives with non-differentiable regularizers. The results obtained are broadly applicable and admit an efficient algorithm with favorable computational complexity. Simulations on realistic setups demonstrate its usefulness in terms of accuracy and computational efficiency in many learning settings.

Submission Length: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=CkbO0xBEsl&nesting=2&sort=date-desc

Changes Since Last Submission: We set the font size and margins in accordance with the TMLR author guide: https://jmlr.org/tmlr/author-guide.html

Assigned Action Editor: ~Mohammad_Emtiyaz_Khan1

Submission Number: 6174