Stateful Dynamics for Training of Binary Activation Recurrent Networks

G. William Chapman IV, Tianyao Patrick Xiao, Corinne Teeter, Sapan Agarwal, Frances S. Chance

Keywords: recurrent network, quantization, spiking neural network, dynamical systems

TL;DR: Local state allows binary activation functions in recurrent neural networks

Abstract:

The excessive energy and memory consumption of neural networks has inspired a recent interest in quantized neural networks.
Due to the discontinuity, training binary neural networks (BNNs) requires modifications or alternatives to standard backpropagation, typically in the form of surrogate gradient descent. Multiple surrogate methods exist for feedforward BNNs; however, their success has been limited when applied to recurrent BNNs, but successful when used in binary-like spiking neural networks (SNNs), which contain intrinsic temporal dynamics. We show that standard binary activation approaches fail to train when applied to layer with explicit recurrent weights, and present a theoretical argument for the necessity of temporal continuity in network behavior. By systematically incorporating mechanisms from SNN models, we find that integrative state enables recurrent binary activation networks to reach similar performance as floating-point approaches, while explicit reset and leakage terms do not affect performance. These results show how spiking units enable the training of binary recurrent neural networks and identify the minimally complex units required to make recurrent binary activations trainable with current surrogate methods.

Primary Area: learning on time series and dynamical systems

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11506