Efficient differentiable programming in a functional array-processing language

Amir Shaikhha, Andrew W Fitzgibbon, Dimitrios Vytiniotis, Simon Peyton Jones

Abstract: We present a system for the automatic differentiation (AD) of a higher-order functional array-processing language. The core functional language underlying this system simultaneously supports both source-to-source forward-mode AD and global optimisations such as loop transformations. In combination, gradient computation with forward-mode AD can be as efficient as reverse mode, and that the Jacobian matrices required for numerical algorithms such as Gauss-Newton and Levenberg-Marquardt can be efficiently computed.