Whitening Convergence Rate of Coupling-based Normalizing Flows

Felix Draxler, Christoph Schnoerr, Ullrich Koethe

Keywords: normalizing flows, generative modeling, RealNVP, theory, maximum likelihood, kullback leibler divergence, invertible neural network, information theory, convergence, coupling block

TL;DR: We show and confirm experimentally an explicit convergence rate for coupling-based normalizing flows for whitening in terms of KL divergence.

Abstract: Coupling-based normalizing flows (e.g. RealNVP) are a popular family of normalizing flow architectures that work surprisingly well in practice. This calls for theoretical understanding. Existing work shows that such flows weakly converge to arbitrary data distributions. However, they make no statement about the stricter convergence criterion used in practice, the maximum likelihood loss. For the first time, we make a quantitative statement about this kind of convergence: We prove that all coupling-based normalizing flows perform whitening of the data distribution (i.e. diagonalize the covariance matrix) and derive corresponding convergence bounds that show a linear convergence rate in the depth of the flow. Numerical experiments demonstrate the implications of our theory and point at open questions.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2210.14032/code)