Squared Neural Families: A New Class of Tractable Density Models

Russell Tsuchida, Cheng Soon Ong, Dino Sejdinovic

Keywords: probabilistic modelling; density estimation; exponential family;

TL;DR: We can compute normalising constants of squared neural networks to construct probability distributions with interesting and useful properties.

Abstract: Flexible models for probability distributions are an essential ingredient in many machine learning tasks. We develop and investigate a new class of probability distributions, which we call a Squared Neural Family (SNEFY), formed by squaring the 2-norm of a neural network and normalising it with respect to a base measure. Following the reasoning similar to the well established connections between infinitely wide neural networks and Gaussian processes, we show that SNEFYs admit closed form normalising constants in many cases of interest, thereby resulting in flexible yet fully tractable density models. SNEFYs strictly generalise classical exponential families, are closed under conditioning, and have tractable marginal distributions. Their utility is illustrated on a variety of density estimation, conditional density estimation, and density estimation with missing data tasks.

Submission Number: 3955