scTree: Discovering Cellular Hierarchies in the Presence of Batch Effects in scRNA-seq Data

Moritz Vandenhirtz, Florian Barkmann, Laura Manduchi, Julia E Vogt, Valentina Boeva

Keywords: hierarchical clustering, batch integration, tree variational autoencoders, variational autoencoders, single-cell RNA sequencing, batch effects

TL;DR: We present a hierarchical clustering method for scRNA-seq data that discovers clusters while simultaneously correcting for batch effects.

Abstract: We propose a novel method, scTree, for single-cell Tree Variational Autoencoders, extending a hierarchical clustering approach to single-cell RNA sequencing data. scTree corrects for batch effects while simultaneously learning a tree-structured data representation. This VAE-based method allows for a more in-depth understanding of complex cellular landscapes independently of the biasing effects of batches. We show empirically on seven datasets that scTree discovers the underlying clusters of the data and the hierarchical relations between them, as well as outperforms established baseline methods across these datasets. Additionally, we visualize the learned trees to better understand the hierarchy and their biological relevance, thus underpinning the importance of integrating batch correction directly into the clustering procedure.

Submission Number: 148