Orthrus: Towards Evolutionary and Functional RNA Foundation Models

Philip Fradkin, Ruian Shi, Keren Isaev, Brendan Frey, Quaid Morris, Leo J Lee, BO WANG

Keywords: Contrastive Learning, Genomics, Self-Supervised Learning, RNA, Orthology, Representation Learning, Alternative Splicing, Few-shot Learning

TL;DR: Contrastive learning for mature mRNA isoforms learns effective representation for RNA property prediction

Abstract: In the face of rapidly accumulating genomic data, our understanding of the RNA regulatory code remains incomplete. Pre-trained genomic foundation models offer an avenue to adapt learned RNA representations to biological prediction tasks. However, existing genomic foundation models are trained using strategies borrowed from textual or visual domains, such as masked language modelling or next token prediction, that do not leverage biological domain knowledge. Here, we introduce Orthrus, a mamba based RNA foundation model pre-trained using a novel self-supervised contrastive learning objective with biological augmentations. Orthrus is trained by maximizing embedding similarity between curated pairs of RNA transcripts, where pairs are formed from splice isoforms of 10 model organisms and transcripts from orthologous genes in 400+ mammalian species from the Zoonomia Project. This training objective results in a latent representation that clusters RNA sequences with functional and evolutionary similarities. We find that the generalized mature RNA isoform representations learned by Orthrus significantly outperform existing genomic foundation models on five mRNA property prediction tasks, and requires only a fraction of fine-tuning data to do so.

Submission Number: 39