A Framework for Neural Topic Modeling with Mutual Information and Group Regularization

Tung Nguyen, Linh Van Ngo, Duc Anh Nguyen, Sang Dinh Viet

Abstract: Recent advances in topic modeling have leveraged pre-trained language models (PLMs) and refined topic-word associations to improve both topic discovery and document topic distributions. However, directly integrating PLMs often leads to higher inference costs, making them less suitable for low-latency applications. At the same time, effectively capturing inter-topic relationships remains a critical yet challenging task. In this paper, we propose NeuroMIG (Neural Topic Modeling with Mutual Information and Group Topic Regularization), a novel framework that addresses both issues. NeuroMIG (1) maximizes mutual information between document topic distributions and PLM embeddings to efficiently incorporate PLM knowledge, and (2) employs Group Topic Regularization based on optimal transport to model interactions among topics. Compatible with a wide range of topic modeling architectures, NeuroMIG significantly improves performance over baselines while preserving efficient inference, as validated by experimental results.

External IDs:dblp:journals/ijon/NguyenNDV25