Time Adaptive Recurrent Neural Network

Anil Kag, Venkatesh Saligrama

Keywords: novel recurrent neural architectures, learning representations of outputs or states

Abstract: We propose a learning method that, dynamically modifies the time-constants of the continuous-time counterpart of a vanilla RNN. The time-constants are modified based on the current observation and hidden state. Our proposal overcomes the issues of RNN trainability, by mitigating exploding and vanishing gradient phenomena based on placing novel constraints on the parameter space, and by suppressing noise in inputs based on pondering over informative inputs to strengthen their contribution in the hidden state. As a result, our method is computationally efficient overcoming overheads of many existing methods that also attempt to improve RNN training. Our RNNs, despite being simpler and having light memory footprint, shows competitive performance against standard LSTMs and baseline RNN models on many benchmark datasets including those that require long-term memory.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: Time-Adaptive RNNs are ODE-inspired RNNs that improve RNN trainability by modulating time-constants in response to current input. This allows for selectively pondering on informative inputs in a way that ensures lossless information propagation.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=LlXV-3J5zL