Universality Theorems for Generative Models

Valentin Khrulkov, Ivan Oseledets

TL;DR: We shot that a wide class of manifolds can be generated by ReLU and sigmoid networks with arbitrary precision.

Abstract: Despite the fact that generative models are extremely successful in practice, the theory underlying this phenomenon is only starting to catch up with practice. In this work we address the question of the universality of generative models: is it true that neural networks can approximate any data manifold arbitrarily well? We provide a positive answer to this question and show that under mild assumptions on the activation function one can always find a feedforward neural network that maps the latent space onto a set located within the specified Hausdorff distance from the desired data manifold. We also prove similar theorems for the case of multiclass generative models and cycle generative models, trained to map samples from one manifold to another and vice versa.

Keywords: generative models, theory, universality, manifolds, differential geometry