Metric Mirages in Cell Embeddings

Hanchen, Jure Leskovec, Aviv Regev

Abstract: Although biological studies increasingly rely on embeddings of single cell profiles, the quality of these embeddings can be challenging to assess. Such evaluations are especially important for avoiding mis- leading biological interpretations, assessing the accuracy of integration methods, and establishing the zero-shot capabilities of foundational models. Here, we posit that current evaluation metrics can be highly misleading. We show this by training a three-layer perceptron, Islander , which outperforms all 11 leading embedding methods on a diverse set of cell atlases, but in fact distorts biological structures, limiting its utility for biological discovery. We then present a metric, scGraph, to flag such distortions. Our work should help learn more robust and reliable cell embeddings.