Contracting Implicit Recurrent Neural Networks: Stable Models with Improved Trainability

Max Revay, Ian Manchester

Abstract: Stability of recurrent models is closely linked with trainability, generalizability and in some applications,safety. Methods that train stable recurrent neural networks, however, do so at a significant cost to expressibility. We propose an implicit model structure that allows for a convex parametrization of stable models using contraction analysis of non-linear systems. Using these stability conditions we propose a new approach to model initialization and then provide a number of empirical results comparing the performance of our proposed model set to previous stable RNNs and vanilla RNNs. By carefully controlling stability in the model, we observe a significant increase in the speed of training and model performance.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1912.10402/code)