RotationOut as a Regularization Method for Neural Network

Kai Hu, Barnabas Poczos

TL;DR: We propose a regularization method for neural network and a noise analysis method

Abstract:

In this paper, we propose a novel regularization method, RotationOut, for neural networks.
Different from Dropout that handles each neuron/channel independently, RotationOut regards its input layer as an entire vector and introduces regularization by randomly rotating the vector.
RotationOut can also be used in convolutional layers and recurrent layers with a small modification.
We further use a noise analysis method to interpret the difference between RotationOut and Dropout in co-adaptation reduction.
Using this method, we also show how to use RotationOut/Dropout together with Batch Normalization.
Extensive experiments in vision and language tasks are conducted to show the effectiveness of the proposed method.
Codes will be available.

Keywords: Neural Network, Regularization

Code: https://github.com/RotationOut/RotationOut

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:1911.07427/code)