Rate-Distortion Auto-Encoders

Luis G. Sanchez Giraldo, Jose C. Principe

Dec 31, 2013 (modified: Dec 31, 2013) ICLR 2014 conference submission readers: everyone
  • Decision: submitted, no decision
  • Abstract: We propose a learning algorithm for auto-encoders based on a rate-distortion objective. Our goal is to minimize the mutual information between the inputs and the outputs of an auto-encoder subject to a fidelity constraint. Minimizing the mutual information acts as a regularization term whereas the fidelity constraint can be understood as a risk functional in the conventional statistical learning setting. The proposed algorithm uses a recently introduced measure of entropy based on infinitely divisible matrices that avoids the plug in estimation of densities. Experiments using over-complete bases show that the auto-encoder learns a regularized input-output map without explicit regularization terms or add-hoc constraints such as tied weights.