Scalable Equilibrium Propagation via Intermediate Error Signals for Deep Convolutional CRNNs

Abstract: Equilibrium Propagation (EP) is a biologically inspired local learning rule first proposed for convergent recurrent neural networks (CRNNs), in which synaptic updates depend only on neuron states from two distinct phases. EP estimates gradients that closely align with those computed by Backpropagation Through Time (BPTT) while significantly reducing computational demands, positioning it as a potential candidate for on-chip training in neuromorphic architectures. However, prior studies on EP have been constrained to shallow architectures, as deeper networks suffer from the vanishing gradient problem, leading to convergence difficulties in both energy minimization and gradient computation. To address the vanishing gradient problem in deep EP networks, we propose a novel EP framework that incorporates intermediate error signals to enhance information flow and convergence of neuron dynamics. This is the first work to integrate knowledge distillation and local error signals into EP, enabling the training of significantly deeper architectures. Our proposed approach achieves state-of-the-art performance on the CIFAR-10 and CIFAR-100 datasets, showcasing its scalability on deep VGG architectures. These results represent a significant advancement in the scalability of EP, paving the way for its application in real-world systems.

Submission Type: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

The explicit changes made to the revised manuscript:
* Re-ran experiments and updated Figures 5a, 5b, and 5e. The y-axes now display the mean absolute values per neuron and per parameter.
* Added a new plot tracking the norm of the difference between the nudged and free states throughout the training process.
* Modified the axis labels and the figure caption for Figure 5 to clearly reflect these new normalized metrics.
* Corrected terminology in Section 2.2.
* Clarified Theorem 3.1.

Assigned Action Editor: ~Christian_Keup1

Submission Number: 7110