Sparsity Programming: Automated Sparsity-Aware Optimizations in Differentiable Programming

Shashi Gowda, Yingbo Ma, Valentin Churavy, Alan Edelman, Christopher Rackauckas

TL;DR: Transforming a program into one that computes the sparsity pattern of its Jacobian or Hessian

Abstract: Previous studies in numerical analysis have shown how the calculation of a Jacobian, Hessian, and their factorizations can be accelerated when their sparsity pattern is known. However, accurate Jacobian and Hessian sparsity patterns cannot be computed numerically, leaving the burden on the user to provide them. As scientific simulations have grown in complexity, the application of differentiable programming for calculating the derivatives of arbitrary programs with respect to parameters has been on the rise, but current methodologies do not automatically detect and make use of sparsity-related accelerations. In this manuscript we develop a method for the accurate and efficient construction of sparsity patterns by transforming an input program into one that computes the sparsity pattern of its Jacobian or Hessian. Our implementation, which we demonstrate on partial differential equations, is a scalable technique for acceleration of automatic differentiation on arbitrarily complex multivariate programs. This demonstrates that dynamic program analysis can be effective in more scenarios than are currently well known in differentiable programming.

Keywords: program transformation, differentiable programming, automatic differentiation, sparsity detection