Max-Affine Spline Insights Into Deep Generative Networks

Randall Balestriero, Sebastien Paris, Richard Baraniuk

Keywords: max affine spline, generative networks, manifold smoothing, dropout, dropconnect, inverse problems, GAN, VAE, multimodal density estimation, elbow method

Abstract:

We connect a large class of Deep Generative Networks (DGNs) with spline operators in order to derive their properties, limitations, and new opportunities.
By characterizing the latent space partition and per-region affine mappings, we relate the manifold dimension and approximation error to the sample size, provide a theoretically motivated ``elbow method'' to infer the intrinsic dimension of the manifold being approximated, study the (not always) beneficial impact of dropout/dropconnect, and demonstrate how the DGN generated piecewise affine surface can be ``smoothed'' to facilitate latent space optimization as used in inverse problems.
The per-region affine subspace defines a local coordinate basis; we provide necessary and sufficient conditions relating those basis vectors with disentanglement and demonstrate how to interpret the DGN learned parameters.
We also derive the output probability density mapped onto the generated manifold in terms of the latent space density, which enables the computation of key statistics such as its Shannon entropy.
This finding also enables us to highlight the source of training instabilities when the target density is multimodal: as the modes gets further apart and/or more concentrated, as the approximant DGN per-region mappings must have increasing singular values leading to large amplitude layer weights and causing training instabilities.

One-sentence Summary: We connect a large class of Deep Generative Networks (DGNs) with spline operators in order to derive their properties, limitations, and new opportunities.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2002.11912/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=sv7Ou2gU-I