Language-Enhanced Representation Learning for Single-Cell Transcriptomics

Jiaqi Yang, Zhiyuan Liu, Yaorui Shi, Qingchuan Zhang, Changhao Nai, Junfeng Fang, Yahui Long, Yang Zhang, Xiang Wang

Keywords: Single-Cell Transcriptomics, Multi-modal LLM

Abstract: Single-cell RNA sequencing (scRNA-seq) offers detailed insights into cellular heterogeneity. Recent advancements leverage single-cell large language models (scLLMs) for effective representation learning. These models focus exclusively on transcriptomic data, neglecting complementary biological knowledge from textual descriptions. To overcome this limitation, we propose scMMGPT, a novel multimodal framework designed for language-enhanced representation learning in single-cell transcriptomics. Unlike existing methods, scMMGPT employs robust cell representation extraction, preserving quantitative gene expression data, and introduces an innovative two-stage pre-training strategy combining discriminative precision with generative flexibility. Extensive experiments across nine datasets and five tasks show that scMMGPT significantly outperforms unimodal and multimodal baselines. Notably, scMMGPT achieves relative improvements of 8.13% for clustering on the COVID-19 dataset, 7.48% for annotation on the Myeloid dataset, and 2.93% for perturbation prediction on the Adamson dataset. Our code is available at https://anonymous.4open.science/r/scMMGPT-2F37/.

Submission Number: 94