Concept decompositions for short text clustering by identifying word communities

Caiyan Jia, Matthew B. Carson, Xiaoyang Wang, Jian Yu

Abstract: Highlights•A new concept decomposition method WordCom is proposed.•It creates concept vectors by identifying semantic word communities from a weighted word co-occurrence network.•It is not only robust to the sparsity of short texts but also overcomes the curse of dimensionality.•It scaling to a large number of short text inputs due to the concept vectors being obtained from term-term space.•Experimental tests have shown that the proposed method outperforms state-of-the-art algorithms.