Enhanced neural network regularization with macro-block dropout

Chanwoo Kim

Keywords: macro block dropout, regularization

Abstract: This paper proposes a new regularization algorithm referred to as macro-block dropout. The overfitting issue has been a difficult problem in training large network models. The dropout technique has proven to be simple yet very effective for regularization by preventing complex co-adaptations on training data. In this work, we observe that in the hidden outputs, the correlations between geometrically close elements are usually stronger than those between distant elements. Motivated by this observation, we define a macro-block that contains multiple elements of the hidden output layer in order to reduce co-adaptations more effectively. Rather than applying dropout to each element, we apply random dropout to each macro-block. In our experiments with image classification tasks on the MNIST and the ImageNet datasets as well as a speech recognition task on the LibriSpeech set, this simple algorithm has shown a quite significant improvement over the conventional dropout approach

One-sentence Summary: In this paper, we propose a macro-block dropout for better regularization.