Variational Gaussian topic model with invertible neural projections

Rui Wang, Deyu Zhou, Yuxuan Xiong, Haiping Huang

Abstract: Neural topic models have triggered a surge of interest in extracting topics from text automatically since they avoid the sophisticated derivations in conventional topic models. However, most of these models only utilize word count information, which will result in low coherent topics. To address this issue, we explore the way to incorporate word relatedness in word embeddings into the topic modeling process and propose the novel Variational Gaussian Topic Model (VaGTM). Based on the variational auto-encoder, the proposed VaGTM models each topic with a multivariate Gaussian in the decoder to group the semantically similar words together and obtains more coherent topics. Moreover, to address the limitation that pre-trained word embedding of topic-associated words does not exactly follows Gaussian, we extend the VaGTM and propose the Variational Gaussian Topic Model with Invertible neural Projections (VaGTM-IP). Extensive experiments have been conducted on three benchmark text corpora, and the experimental results show that VaGTM and VaGTM-IP outperform several competitive baselines in terms of four topic coherence metrics. Besides, the proposed approaches could also mine topic correlations between extracted topics with topic-associated Gaussian distributions.