Empirical Study of Rectifier and Dropout in Feedforward Neural Networks

Sungpil Woo, Sunhwan Lim, Daeyoung Kim

Abstract: In this paper, we analyze the unique sparsity properties of dropout, rectified activation functions, and other activation functions through simple experiments. We explore how the distinctive membership functions of each contribute to these sparsity characteristics.