Contextualized Topic Coherence Metrics

Anonymous

Abstract: This article proposes LLM-based topic coherence metrics inspired by standard human topic evaluations, in a family of metrics called Contextualized Topic Coherence (CTC). These metrics allow human-centered evaluation of coherence while maintaining the efficiency of automated methods. We evaluate CTC relative to five metrics and discovered that it outperforms automated topic coherence methods on seven topic models. Notably, CTC aligns with human evaluation and demonstrates excellent performance with short documents, and is not susceptible to meaningless but high-scoring topics.

Paper Type: long

Research Area: Information Retrieval and Text Mining

Contribution Types: Model analysis & interpretability

Languages Studied: English

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.