Towards foundation models that learn across biological scales

ICML 2025 Workshop FM4LS Submission1 Authors

Keywords: machine learning, biology, scRNAseq, proteins, DNA, RNA, multi-scale, learning, representation, foundation, model, AI, life

TL;DR: how we can build models that learn hierarchically, from atoms to full organisms

Abstract: We have reached a point where many bio foundation models exist across 4 different scales, from molecules to molecular chains, cells, and tissues. However, while related in many ways, these models do not yet bridge these scales. We present a framework and architecture called Xpressor that enables cross-scale learning by (1) using a novel cross-attention mechanism to compress high-dimensional gene representations into lower-dimensional cell-state vectors, and (2) implementing a multi-scale fine-tuning approach that allows cell models to leverage and adapt protein-level representations. Using a cell Foundation Model as an example, we demonstrate that our architecture improves model performance across multiple tasks, including cell-type prediction (+12%) and embedding quality (+8%). Together, these advances represent first steps toward models that can understand and bridge different scales of biological organization.

Submission Number: 1