Generating and decoding methylated DNA with a Human Epigenetic Foundation Model

Pouya Niki, Christoforos Nalmpantis, Javkhlan-Ochir Ganbat, Donal Byrne, Husam Babikir, Anjeet Jhutty, Andrey Karailiev, Pooja Kathail, Luca Giacomoni, Francisco M Martín-Zamora, Netanel Loyfer, Khaled Kamal Saab, Hannah Madan, Jonathan C M Wan, Ravi Solanki

Track: Full / long paper (5-8 pages)

Keywords: AI, biology, genomics, methylation, epigenetics, pleiades, neuron, cell type, brain, transformer, language model, deconvolution, cell free dna

TL;DR: Epigenetic Foundation Model with Clinical and Biological Applications

Abstract: Gene expression in humans is regulated beyond the four-letter genetic code; cytosine methylation programs cell identity and regulates expression in response to environmental cues. We present Pleiades, a series of whole-epigenome foundation models (90M/600M/7B) trained on 1.9T tokens of methylated and unmethylated human DNA, establishing a new paradigm beyond the modeling of pure DNA sequences. Pleiades achieves state-of-the-art performance compared to leading DNA foundation models on human genomic annotation tasks, such as predicting histone modifications and gene regulatory elements; notably, we find that scaling model size yields consistent gains across all tasks, with the 7B model outperforming both smaller variants and DNA-only baselines. Finally, we show that Pleiades supports a number of cell-free DNA (cfDNA) tasks, opening the door to a new era of direct clinical application of biological foundation models via cfDNA.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 13