Saturating Auto-Encoder

Ross Goroshin, Yann LeCun

Jan 18, 2013 (modified: Jan 18, 2013) ICLR 2013 conference submission readers: everyone
  • Decision: conferencePoster-iclr2013-conference
  • Abstract: We introduce a simple new regularizer for auto-encoders whose hidden-unit activation functions contain at least one zero-gradient (saturated) region. This regularizer explicitly encourages activations in the saturated region(s) of the corresponding activation function. We call these Saturating Auto-Encoders (SATAE). We show that the saturation regularizer explicitly limits the SATAE's ability to reconstruct inputs which are not near the data manifold. Furthermore, we show that a wide variety of features can be learned when different activation functions are used. Finally, connections are established with the Contractive and Sparse Auto-Encoders.

Loading