Discrete Flows: Invertible Generative Models of Discrete Data

Dustin Tran, Keyon Vafa, Kumar Agrawal, Laurent Dinh, Ben Poole

Keywords: discrete data, normalizing flows, autoregressive, realnvp

TL;DR: We extend autoregressive flows and RealNVP to discrete data.

Abstract: While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown. In this paper, we show that flows can in fact be extended to discrete events---and under a simple change-of-variables formula not requiring log-determinant-Jacobian computations. Discrete flows have numerous applications. We display proofs of concept under 2 flow architectures: discrete autoregressive flows enable bidirectionality, allowing for example tokens in text to depend on both left-to-right and right-to-left contexts in an exact language model; and discrete bipartite flows (i.e., with layer structure from RealNVP) enable parallel generation such as exact nonautoregressive text modeling.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:1905.10347/code)