Generalization of the Change of Variables Formula with Applications to Residual Flows

Niklas Koenen, Marvin N. Wright, Peter Maass, Jens Behrmann

Keywords: Machine Learning, ICML, Change of Variables, Generalization of Flows, Residual Flows, Measure Theory

Abstract: Normalizing flows leverage the Change of Variables Formula (CVF) to define flexible density models. Yet, the requirement of smooth transformations (diffeomorphisms) in the CVF poses a significant challenge in the construction of these models. To enlarge the design space of flows, we introduce $\mathcal{L}$-diffeomorphisms as generalized transformations which may violate these requirements on zero Lebesgue-measure sets. This relaxation allows e.g. the use of non-smooth activation functions such as ReLU. Finally, we apply the obtained results to planar, radial, and contractive residual flows.