Low-rank passthrough neural networks

Antonio Valerio Miceli Barone

Abstract: Deep learning consists in training neural networks to perform computations that sequentially unfold in many steps over a time dimension or an intrinsic depth dimension. For large depths, this is usually accomplished by specialized network architectures that are designed to mitigate the vanishing gradient problem, e.g. LSTMs, GRUs, Highway Networks and Deep Residual Networks, which are based on a single structural principle: the state passthrough. We observe that these "Passthrough Networks" architectures enable the decoupling of the network state size from the number of parameters of the network, a possibility that is exploited in some recent works but not thoroughly explored. In this work we propose simple, yet effective, low-rank and low-rank plus diagonal matrix parametrizations for Passthrough Networks which exploit this decoupling property, reducing the data complexity and memory requirements of the network while preserving its memory capacity. We present competitive experimental results on several tasks, including a near state of the art result on sequential randomly-permuted MNIST classification, a hard task on natural data.

TL;DR: Describe low-rank and low-rank plus diagonal parametrizations for Highway Neural Networks, GRUs and other kinds of passthrough neural networks. Present competitive experimental results.

Conflicts: ed.ac.uk, unipi.it

Keywords: Deep learning

Community Implementations: [ 9 code implementations](https://www.catalyzex.com/paper/arxiv:1603.03116/code)