Provable Guarantees on Learning Hierarchical Generative Models with Deep CNNs

Eran Malach, Shai Shalev-Shwartz

Abstract:

Learning deep networks is computationally hard in the general case. To show any positive theoretical results, one must make assumptions on the data distribution. Current theoretical works often make assumptions that are very far from describing real data, like sampling from Gaussian distribution or linear separability of the data. We describe an algorithm that learns convolutional neural network,
assuming the data is sampled from a deep generative model that generates images level by level,
where lower resolution images correspond to latent semantic classes. We analyze the convergence rate of our algorithm assuming the data is indeed generated according to this model (as well as
additional assumptions). While we do not pretend to claim that the assumptions are realistic for natural images, we do believe that they capture some true properties of real data. Furthermore, we show that on CIFAR-10, the algorithm we analyze achieves results in the same ballpark with vanilla convolutional neural networks that are trained with SGD.

TL;DR: A generative model for deep CNNs with provable theoretical guarantees that actually works

Keywords: deep learning, theory