Mutual Information Dropout: Mutual Information Can Be All You Need

Zichen Song, Shan Ma

Abstract: Dropout is a powerful way for preventing model overfitting. However, it is inefficient due to it randomly ignoring some neurons. Although there are many ways on Dropout, they are still either inefficient on improving generalization ability or not effective enough. In this paper, we propose Mutual Information Dropout, which is an efficient Dropout based on dropping neurons with low mutual information. In Mutual Information Dropout, instead of randomly ignoring some neurons, we first evaluated the mutual information of neurons to dropout with mutual information below a certain threshold. In this way, Mutual Information Dropout can achieve effective improving generalization ability with evaluate neurons. Extensive experiments on Three datasets show that Mutual Information Dropout is much more efficient than many existing Dropout and can meanwhile achieve comparable or even better generalization ability.