Resonator-Gated RNNs

Robert Deibel, Shahram Eivazi, Martin V. Butz, Sebastian Otte

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: sequence learning, RNN, LSTM, resonators, time series, ECG, MNIST, speech commands

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Sequence learning tasks frequently involve data with repetitive and periodic temporal patterns. Detecting these patterns is essential for accurate predictions and informed decision-making in various domains. There is, however, still huge potential in augmenting sequence learning algorithms in this regard. In RNN-based sequence learning, gated RNNs, such as long short-term memory networks (LSTMs)
and gated recurrent units (GRUs), are the de facto standard. While adept at capturing longer-term dependencies, gated RNNs still sometimes struggle with periodic data components, because their gating mechanism is designed to prioritize retaining static relevant information. As a result, these networks often challenged by periodicity in the data. We present a novel memory unit that incorporates a simple resonator circuit. The resonator facilitates the recognition of periodic data patterns, focusing on data-specific time scales and respective frequencies. Moreover, it enables the forward propagation of information through resonating dynamics while stably channeling the gradient backwards. We show that our resonator-gated RNN (RG-RNN) accelerates the training convergence on multiple sequence classifications tasks. Moreover, it significantly outperforms vanilla LSTMs on three out of four benchmark tasks in terms of accuracy. We conclude that resonator-based gating offers a new inductive bias to gated RNNs, focusing learning on the detection and processing of periodic data patterns.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7686