A Heavy-Tailed Algebra for Probabilistic Programming

Feynman T. Liang, Liam Hodgkinson, Michael W. Mahoney

Keywords: probabilistic programming, static analysis, heavy tails, monte carlo, mcmc, variational inference

TL;DR: Develops an algebra for analyzing tails and operationalizes it in a PPL compiler to automatically inform the design of density approximator with provably calibrated tails.

Abstract: Despite the successes of probabilistic models based on passing noise through neural networks, recent work has identified that such methods often fail to capture tail behavior accurately---unless the tails of the base distribution are appropriately calibrated. To overcome this deficiency, we propose a systematic approach for analyzing the tails of random variables, and we illustrate how this approach can be used during the static analysis (before drawing samples) pass of a probabilistic programming language (PPL) compiler. To characterize how the tails change under various operations, we develop an algebra which acts on a three-parameter family of tail asymptotics and which is based on the generalized Gamma distribution. Our algebraic operations are closed under addition and multiplication; they are capable of distinguishing sub-Gaussians with differing scales; and they handle ratios sufficiently well to reproduce the tails of most important statistical distributions directly from their definitions. Our empirical results confirm that inference algorithms that leverage our heavy-tailed algebra attain superior performance across a number of density modeling and variational inference (VI) tasks.

Submission Number: 1689