Adaptive-saturated RNN: Remember more with less instability

Khoi Minh Nguyen-Duy, Quang Pham, Binh T. Nguyen

Keywords: recurrent neural networks, vanishing gradient, memory capacity, adaptive saturation

TL;DR: A novel variant of Tanh RNN that is easily trained without the vanishing gradient problem.

Abstract: Orthogonal parameterization is a compelling solution to the vanishing gradient problem (VGP) in recurrent neural networks (RNNs). With orthogonal parameters and non-saturated activation functions, gradients in such models are constrained to unit norms. On the other hand, although the traditional vanilla RNNs are seen to have higher memory capacity, they suffer from the VGP and perform badly in many applications. This work proposes Adaptive-Saturated RNNs (asRNN), a variant that dynamically adjusts its saturation level between the two mentioned approaches. Consequently, asRNN enjoys both the capacity of a vanilla RNN and the training stability of orthogonal RNNs. Our experiments show encouraging results of asRNN on challenging sequence learning benchmarks compared to several strong competitors. The research code is accessible at https://github.com/ndminhkhoi46/asRNN/.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2304.11790/code)