Fast Inference and Learning for Modeling Documents with a Deep Boltzmann Machine

Nitish Srivastava, Ruslan Salakhutdinov, Geoffrey Hinton

Decision: conferencePoster

Abstract:

We introduce a type of Deep Boltzmann Machine (DBM) that is suitable for
extracting distributed semantic representations from a large unstructured
collection of documents. We propose an approximate inference method that interacts with learning in a way that makes it possible to train the DBM more efficiently than previously proposed methods. Even though the model has two hidden layers, it can be trained just as efficiently as a standard Restricted Boltzmann Machine. Our experiments show that the model assigns better log probability to unseen data than the Replicated Softmax model. Features extracted from our model outperform LDA, Replicated Softmax, and DocNADE models on document retrieval and document classification tasks.