Gradient-based Dynamic Dropout

Limin Ma, Xingyuan Chen, Bin Wang, Peng Jin

Abstract: With the development of deep neural networks, in order to deal with overfitting, many researchers have proposed various methods. Dropout is a regularization technique and is often used in deep neural networks, especially in the field of image classification, so that the classification accuracy can be improved. However, these methods rarely consider that the change of parameters can affect the role of neurons in neural networks. In this paper, we propose a dynamic dropout to use a parameter gradient-based method, which utilizes the cumulative sum of the absolute value of the gradient of each hidden unit corresponding to all parameters. Specifically, each neuron has a corresponding gradient under the circumstances, and then we only do dropout on neurons which have large gradient value. The purpose is to let neurons with small gradient value learn more information. We evaluated the effectiveness of Dynamic Dropout experiments on multiple datasets, including MNIST, CIFAR-10, and CIFAR-100, the models are convolutional neural networks.

External IDs:dblp:conf/cis/MaCWJ21