Expanding Horizons in Short Text Analysis: Integrating LLMs and VAEs for Enhanced Topic Modeling

Anonymous

Abstract: Topic models are one of the compelling methods for discovering latent semantics in a document collection. However, it assumes that a document has sufficient co-occurrence information to be effective. However, in short texts, co-occurrence information is minimal, which results in feature sparsity in document representation. Therefore, existing topic models (probabilistic or neural) mostly fail to mine patterns from them to generate coherent topics. In this paper, we take a new approach to short-text topic modeling to address the data-sparsity issue by extending short text into longer sequences using large language models (LLMs) and decoding topics using a variational autoencoder (VAE). We observe that our model can substantially improve the performance of short-text topic modeling. Extensive experiments on multiple real-world datasets under extreme data sparsity scenarios show that our models can generate high-quality topics that outperform state-of-the-art models.

Paper Type: long

Research Area: Information Extraction

Languages Studied: English

Preprint Status: There is a non-anonymous preprint (URL specified in the next question).

A1: yes

A2: no

A3: yes

B: no

B1: yes

C: yes

D: yes

E: no