Contrastive Learning with Simplicial Convolutional Networks for Short-Text Classification

Huang Liang, Benedict Lee, Daniel Hui Loong Ng, Kelin Xia

Abstract: Text classification is a fundamental task in Natural Language Processing (NLP). Short text classification has recently captured much attention due to its increased amount from various sources with limited labels and its inherent challenges for its sparsity in words and semantics. Recent studies have adopted self-supervised contrastive learning across different representations to improve performance. However, most of the current models face several challenges. Firstly, the augmentation step might not be able to generate positive and negative samples that are semantically similar and dissimilar to the anchor respectively. Secondly, the text data could be enhanced with external auxiliary information that might introduce noise to the sparse text data. In addition, they are limited in capturing higher-order information such as group-wise interactions. In this work, we propose a novel document simplicial complex construction based on text data for a higher-order message-passing mechanism. We enhance the short text classification performance by contrasting the structural representation with the sequential representation generated by the transformer mechanism for improved outcomes and mitigated issues. The proposed framework, Contrastive Learning with Simplicial Convolutional Networks (C-SCN), leverages the expressive power of graph neural networks, models higher-order information beyond pair-wise relations and enriches features through contrastive learning. Experimental results on four benchmark datasets demonstrate the capability of C-SCN to outperform existing models in analysing sequential and complex short-text data.

Lay Summary: Classifying short texts is challenging due to their limited word count, sparse meaning, and limited labels. Recent methods utilise contrastive learning, which involves comparing similar and dissimilar text samples, to improve performance. However, many struggle to generate meaningful comparisons, add noisy external data, or miss higher-level patterns. We propose C-SCN, a novel model that combines contrastive learning with simplicial convolutional networks to capture deeper relationships in text. Unlike traditional methods, C-SCN represents documents as structured networks in simplicial complexes, enabling richer analysis beyond word pairs. It contrasts these structures with transformer-generated text embeddings, improving accuracy while reducing noise. Tests on four benchmark datasets show C-SCN outperforms existing models, proving its strength in handling short, complex texts. This approach could enhance applications like social media analysis, customer feedback sorting, or real-time content tagging.

Primary Area: General Machine Learning->Representation Learning

Keywords: Topological Deep Learning, Representation Learning, Message-Passing Mechanism

Submission Number: 14613