Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
David Krueger, Tegan Maharaj, Janos Kramar, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, Christopher Pal
Nov 04, 2016 (modified: Feb 11, 2017)ICLR 2017 conference submissionreaders: everyone
Abstract:We propose zoneout, a novel method for regularizing RNNs.
At each timestep, zoneout stochastically forces some hidden units to maintain their previous values.
Like dropout, zoneout uses random noise to train a pseudo-ensemble, improving generalization.
But by preserving instead of dropping hidden units, gradient information and state information are more readily propagated through time, as in feedforward stochastic depth networks.
We perform an empirical investigation of various RNN regularizers, and find that zoneout gives significant performance improvements across tasks. We achieve competitive results with relatively simple models in character- and word-level language modelling on the Penn Treebank and Text8 datasets, and combining with recurrent batch normalization yields state-of-the-art results on permuted sequential MNIST.
TL;DR:Zoneout is like dropout (for RNNs) but uses identity masks instead of zero masks
Enter your feedback below and we'll get back to you as soon as possible.