Noisy Activation Functions

Çaglar Gülçehre, Marcin Moczulski, Misha Denil, Yoshua Bengio

Abstract: Common nonlinear activation functions used in neural networks can cause training difficulties due to the saturation behavior of the activation function, which may hide dependencies that are not vis...