Residual Echo State Networks: Residual recurrent neural networks with stable dynamics and fast learning

Andrea Ceni, Claudio Gallicchio

Abstract: Highlights•We study residual (skip) connections in the context of RNNs in the temporal dimension.•Temporal residual connections enable long-term processing of time series.•Orthogonal skip connections allow to drive RNNs to the edge-of-stability in an approximate dynamical isometry regime.•Experiments on memory, forecasting, and classification tasks show the benefits of temporal residual connections in RNNs.