Adjoint Method: The Connection between Analog-based Equilibrium Propagation Architectures and Neural ODEs

Mohamed Watfa, Alberto Garcia-Ortiz

Keywords: neural networks, optimizaton, equilibrium propagation, neural ordinary differential equation, hardware design, analog circuit

TL;DR: The paper demonstrates that the adjoint method serves as a key framework linking Equilibrium Propagation in Analog Neural Networks to Neural Ordinary Differential Equations, offering key insights into developing efficient hardware solutions.

Abstract:

Analog neural networks (ANNs) hold significant potential for substantial
reductions in power consumption in modern neural networks, particularly when
employing the increasingly popular Energy-Based Models (EBMs) in tandem with
the local Equilibrium Propagation (EP) training algorithm. This paper analyzes
the relationship between this family of ANNs and the concept of Neural Ordinary
Differential Equations (Neural ODEs). Using the adjoint method, we formally
demonstrate that ANN-EP can be derived from Neural ODEs by constraining the
differential equations to those with a steady-state response. This finding
opens avenues for the ANN-EP community to extend ANNs to non-steady-state
scenarios. Additionally, it provides an efficient setting for NN-ODEs that
significantly reduces the training cost.

Submission Number: 11