Discrete Flows: Invertible Generative Models of Discrete DataDownload PDF

27 Mar 2019 (modified: 02 Jul 2019)ICLR 2019 Workshop DeepGenStruct Blind SubmissionReaders: Everyone
  • Keywords: discrete data, normalizing flows, autoregressive, realnvp
  • TL;DR: We extend autoregressive flows and RealNVP to discrete data.
  • Abstract: While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown. In this paper, we show that flows can in fact be extended to discrete events---and under a simple change-of-variables formula not requiring log-determinant-Jacobian computations. Discrete flows have numerous applications. We display proofs of concept under 2 flow architectures: discrete autoregressive flows enable bidirectionality, allowing for example tokens in text to depend on both left-to-right and right-to-left contexts in an exact language model; and discrete bipartite flows (i.e., with layer structure from RealNVP) enable parallel generation such as exact nonautoregressive text modeling.
3 Replies