Automatic variational inference with cascading flows

Luca Ambrogioni, Gianluigi Silvestri, Marcel van Gerven

Keywords: Variational Inference, Normalizing Flows

TL;DR: A novel Normalizing Flow with gating mechanism to perform automatic structured variational inference.

Abstract:

The automation of probabilistic reasoning is one
of the primary aims of machine learning. Recently, the confluence of variational inference and
deep learning has led to powerful and flexible automatic inference methods that can be trained by
stochastic gradient descent. In particular, normalizing flows are highly parameterized deep models
that can fit arbitrarily complex posterior densities.
However, normalizing flows struggle in highly
structured probabilistic programs as they need
to relearn the forward-pass of the program. Automatic structured variational inference (ASVI)
remedies this problem by constructing variational
programs that embed the forward-pass. Here, we
combine the flexibility of normalizing flows and
the prior-embedding property of ASVI in a new
family of variational programs, which we named
cascading flows. A cascading flows program interposes a newly designed highway flow architecture in between the conditional distributions
of the prior program such as to steer it toward
the observed data. These programs can be constructed automatically from an input probabilistic program and can also be amortized automatically. We evaluate the performance of the new
variational programs in a series of structured inference problems. We find that cascading flows
have much higher performance than both normalizing flows and ASVI in a large set of structured
inference problems.