Inference and Introspection in Deep Generative Models of Sparse Data

Rahul G. Krishnan, Matthew Hoffman

Abstract: Deep generative models such as deep latent Gaussian models (DLGMs) are powerful and popular density estimators. However, they have been applied almost exclusively to dense data such as images; DLGMs are rarely applied to sparse, high-dimensional integer data such as word counts or product ratings. One reason is that the standard training procedures find poor local optima when applied to such data. We propose two techniques that alleviate this problem, significantly improving our ability to fit DLGMs to sparse, high-dimensional data. Having fit these models, we are faced with another challenge: how to use and interpret the representation that we have learned? To that end, we propose a method that extracts distributed representations of features via a simple linearization of the model.

TL;DR: We study two techniques to improve learning in deep generative models on sparse, high-dimensional text data. We also propose an algorithmic tool to visualize and introspect arbitrarily deep learned models.

Conflicts: cs.nyu.edu, adobe.com, columbia.edu

Keywords: Unsupervised Learning, Deep learning