Learning Discrete Distributions by Dequantization

Emiel Hoogeboom, Taco Cohen, Jakub Mikolaj Tomczak

Keywords: dequantization, normalizing flows

TL;DR: Dequantization can be probabilistically grounded using a latent variable model with variational inference

Abstract: Media is generally stored digitally and is therefore discrete. Many successful deep distribution models in deep learning learn a density, i.e., the distribution of a continuous random variable. Naïve optimization on discrete data leads to arbitrarily high likelihoods, and instead, it has become standard practice to add noise to datapoints. In this paper, we present a general framework for dequantization via latent variable modelling. In this framework, we are able to recover existing dequantization schemes as special cases, and we are able to derive natural extensions from variational inference literature. We investigate two unexplored directions for dequantization: More sophisticated inference objectives, based on importance-weighting (iw) and Rényi variational inference. In addition, we analyze dequantization for different types of distributions, and show that autoregressive dequantization achieves 3.06 bits per dimension in negative log-likelihood on CIFAR10.