M-FAC: Efficient Matrix-Free Approximations of Second-Order Information

Elias Frantar, Eldar Kurtic, Dan Alistarh

Keywords: second-order methods, Fisher information matrix, efficient numerical methods, sparsity, optimization, pruning, matrix-free algorithms

TL;DR: We provide efficient matrix-free approximations for inverse-Hessian vector products, with applications in optimization and neural network pruning.

Abstract:

Efficiently approximating local curvature information of the loss function is a useful tool for the optimization and compression of deep neural networks.
Yet, most existing methods to approximate second-order information have high computational or storage costs, limiting their practicality.
In this work, we investigate matrix-free approaches for estimating Inverse-Hessian Vector Products (IHVPs) for the case when the Hessian can be approximated as a sum of rank-one matrices, as in the classic approximation of the Hessian by the empirical Fisher matrix.
The first algorithm we propose is tailored towards network compression and can compute the IHVP for dimension $d$ given a fixed set of $m$ rank-one matrices using $O(dm^2)$ precomputation, $O(dm)$ cost for computing the IHVP and query cost $O(m)$ for computing any single element of the inverse Hessian approximation.
The second algorithm targets an optimization setting, where we wish to compute the product between the inverse Hessian, estimated over a sliding window of optimization steps, and a given gradient direction. We give an algorithm with cost $O(dm + m^2)$ for computing the IHVP and $O(dm + m^3)$ for adding or removing any gradient from the sliding window.
We show that both algorithms yield competitive results for network pruning and optimization, respectively, with significantly lower computational overhead relative to existing second-order methods.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/IST-DASLab/M-FAC