Recursive Construction of Stable Assemblies of Recurrent Neural Networks

Leo Kozachkov, Michaela M Ennis, Jean-Jacques Slotine

Keywords: stability, deep learning, RNN, combinations, modularity, sparsity, negative feedback, sequence learning, contraction analysis

Abstract: Advanced applications of modern machine learning will likely involve combinations of trained networks, as are already used in spectacular systems such as DeepMind's AlphaGo. Recursively building such combinations in an effective and stable fashion while also allowing for continual refinement of the individual networks - as nature does for biological networks - will require new analysis tools. This paper takes a step in this direction by establishing contraction properties of broad classes of nonlinear recurrent networks and neural ODEs, and showing how these quantified properties allow in turn to recursively construct stable networks of networks in a systematic fashion. The results can also be used to stably combine recurrent networks and physical systems with quantified contraction properties. Similarly, they may be applied to modular computational models of cognition. We perform experiments with these combined networks on benchmark sequential tasks (e.g permuted sequential MNIST) to demonstrate their capacity for processing information across a long timescale in a provably stable manner.

One-sentence Summary: Provably stable RNNs perform near-SOTA on benchmark sequential image tasks with few trainable parameters, by leveraging combination properties and sparsity.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2106.08928/code)