BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks

Frederikke Isa Marin, Felix Teufel, Marc Horlacher, Dennis Madsen, Dennis Pultz, Ole Winther, Wouter Boomsma

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Biological sequence analysis, enhancer annotation, gene finding, gene annotation, Language model, genome modelling, benchmark, LLM, embeddings, representations, DNA

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A dataset and downstream tasks for benchmarking emerging DNA language models with realistic and biologically meaningful tasks

Abstract:

The genome sequence contains the blueprint for governing cellular processes.
While the availability of genomes has vastly increased over the last decades, experimental annotation of the various functional, non-coding and regulatory elements encoded in the DNA sequence remains both expensive and challenging. This has sparked interest in unsupervised language modeling of genomic DNA, a paradigm that has seen great success for protein sequence data.
Although various DNA language models have been proposed, evaluation tasks often differ between individual works, and might not fully recapitulate the fundamental challenges of genome annotation, including the length, scale and sparsity of the data. In this study, we introduce **BEND**, a **BEN**chmark for **D**NA language models, featuring
a collection of realistic and biologically meaningful downstream tasks defined on the human genome.
We find that embeddings from current DNA LMs can approach performance of expert methods on some tasks, but only capture limited information about long-range features.
BEND is available at https://github.com/frederikkemarin/BEND.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Primary Area: datasets and benchmarks

Submission Number: 5937