Adjoint Method: The Connection between Analog-based Equilibrium Propagation Architectures and Neural ODEs

Published: 01 Nov 2023, Last Modified: 22 Dec 2023MLNCP PosterEveryoneRevisionsBibTeX
Keywords: neural networks, optimizaton, equilibrium propagation, neural ordinary differential equation, hardware design, analog circuit
TL;DR: The paper demonstrates that the adjoint method serves as a key framework linking Equilibrium Propagation in Analog Neural Networks to Neural Ordinary Differential Equations, offering key insights into developing efficient hardware solutions.
Abstract: Analog neural networks (ANNs) hold significant potential for substantial reductions in power consumption in modern neural networks, particularly when employing the increasingly popular Energy-Based Models (EBMs) in tandem with the local Equilibrium Propagation (EP) training algorithm. This paper analyzes the relationship between this family of ANNs and the concept of Neural Ordinary Differential Equations (Neural ODEs). Using the adjoint method, we formally demonstrate that ANN-EP can be derived from Neural ODEs by constraining the differential equations to those with a steady-state response. This finding opens avenues for the ANN-EP community to extend ANNs to non-steady-state scenarios. Additionally, it provides an efficient setting for NN-ODEs that significantly reduces the training cost.
Submission Number: 11