Dropout: Explicit Forms and Capacity Control

Anonymous

Sep 25, 2019 Blind Submission readers: everyone Show Bibtex
  • Abstract: We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix sensing, where it induces a data-dependent regularizer that, in expectation, equals the weighted trace-norm of the product of the factors. In deep learning, we show that the data-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks. We evaluate our theoretical findings on real-world datasets, including MovieLens, Fashion MNIST, and CIFAR-10.
  • Code: https://www.dropbox.com/s/inptu0exz9iz4cb/c75_drop.py?dl=0
0 Replies

Loading