Automatic Differentiation: Inverse Accumulation Mode

Jeffrey Mark Siskind

Keywords: Newton step

TL;DR: extending forward and reverse AD to compute product of Jacobian inverses with vectors on the left or right

Abstract:

We show that, under certain circumstances, it is possible to
automatically compute Jacobian-inverse-vector and
Jacobian-inverse-transpose-vector products about as efficiently as
Jacobian-vector and Jacobian-transpose-vector products.
The key insight is to notice that the Jacobian corresponding to the
use of one primitive arithmetic operator is of a form whose sparsity
is invariant to inversion.
This technique has the potential to allow the efficient direct
calculation of Newton steps.