HyperHELM: Hyperbolic Hierarchy Encoding for mRNA Language Modeling

Max van Spengler, Artem Moskalev, Tommaso Mansi, Mangal Prakash, Rui Liao

Keywords: Hyperbolic embeddings, Hierarchical modeling, mRNA language models, mRNA property prediction

TL;DR: We guide mRNA language modeling by embedding the codon hierarchy in hyperbolic space and using the resulting embedding as a classifier during masked language modeling pre-training.

Abstract: Language models are increasingly applied to biological sequences like proteins and mRNA, yet their default Euclidean geometry may mismatch the hierarchical structures inherent to biological data. While hyperbolic geometry provides a better alternative for accommodating hierarchical data, it has yet to find a way into language modeling for mRNA sequences. In this work, we introduce HyperHELM, a framework that implements masked language model pre-training in hyperbolic space for mRNA sequences. Using a hybrid design with hyperbolic layers atop Euclidean backbone, HyperHELM aligns learned representations with the biological hierarchy defined by the relationship between mRNA and amino acids. Across multiple multi-species datasets, it outperforms Euclidean baselines on 9 out of 10 tasks involving property prediction, with 10\% improvement on average, and excels in out-of-distribution generalization to long and low-GC content sequences; for antibody region annotation, it surpasses hierarchy-aware Euclidean models by 3\% in annotation accuracy. Our results highlight hyperbolic geometry as an effective inductive bias for hierarchical language modeling of mRNA sequences.

Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)

Submission Number: 2449