Automatic Differentiation in Myia

Olivier Breuleux, Bart van Merriënboer

Abstract: Automatic differentiation is an essential feature of machine learning frameworks.However, its implementation in existing frameworks often has limitations. In dataflow programming frameworks such as Theano or TensorFlow the representation used makes supporting higher-order gradients difficult. On the other hand, operator overloading frameworks such as PyTorch are flexible, but do not lend themselves well to optimization. With Myia, we attempt to have the best of both worlds: Building on the work by Pearlmutter and Siskind we implement a first-order gradient operator for a subset of the Python programming language.

Keywords: automatic differentiation, programming languages, machine learning, deep learning