Differentiate Everything with a Reversible Domain-Specific Language

JinGuo Liu, Taine Zhao

Keywords: Reversible computing, automatic differentiation, Julia

Abstract:

Reverse-mode automatic differentiation (AD) suffers from the issue of having too much space overhead to trace back intermediate computational states for backpropagation.
The traditional method to trace back states is called checkpointing that stores intermediate states into a global stack and restore state through either stack pop or re-computing.
The overhead of stack manipulations and re-computing makes the general purposed (or not tensor-based) AD engines unable to meet many industrial needs.
Instead of checkpointing, we propose to use reverse computing to trace back states by designing and implementing a reversible programming eDSL, where a program can be executed bi-directionally without implicit stack operations. The absence of implicit stack operations makes the program compatible with existing compiler features, including utilizing existing optimization passes and compiling the code as GPU kernels.
We implement AD for sparse matrix operations and some machine learning applications to show that our framework has state-of-the-art performance.

One-sentence Summary: Design a reversible eDSL in Julia with native high performance AD support.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=1b5vhgo0lC