The Differentiable Curry

Dimitrios Vytiniotis, Dan Belov, Richard Wei, Gordon Plotkin, Martin Abadi

Keywords: automatic differentiation, higher-order programs

Abstract: We revisit the automatic differentiation (AD) of programs that contain higher-order functions, in a statically typed setting. Our presentation builds on a recent formulation of AD based on categorical combinators, and shows how that formulation can be extended to higher-order functions via two different notions of differentiable currying and evaluation. We present these alternative implementations, and justify their correctness by means of showing that AD yields equivalent back-propagators for forward-equivalent programs, even in the presence of higher-order features. Higher-order functions complicate the definition of equivalence for back-propagators compared to simpler forms of program equivalence.