DNAMotifTokenizer: Towards Biologically Informed Tokenization of Genomic Sequences

ICLR 2026 Conference Submission20888 Authors

Keywords: DNA language model, Tokenization, Genomics, Sequence motifs

TL;DR: A novel DNA tokenizer designed to incorporate prior knowledge of DNA sequence motifs for better genomic representation

Abstract: DNA language models have advanced genomics, but their downstream performance varies widely due to differences in tokenization, pretraining data, and architecture. We argue that a major bottleneck lies in tokenizing sparse and unevenly distributed DNA sequence motifs, which are critical for accurate and interpretable models. To investigate, we systematically benchmark k-mer and Byte-Pair Encoding (BPE) tokenizers under controlled pretraining, evaluating across multiple downstream tasks from five datasets. We find that tokenizer choice induces task-specific trade-offs, and that vocabulary size and training data strongly influence the biological knowledge captured. Notably, BPE tokenizers achieve strong performance when trained on smaller but biologically significant data. Building on these insights, we introduce DNAMotifTokenizer, which directly incorporates domain knowledge of DNA sequence motifs into the tokenization process. DNAMotifTokenizer consistently outperforms BPE across diverse benchmarks, demonstrating that knowledge-infused tokenization is crucial for learning powerful, interpretable, and generalizable genomic representations.

Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)

Submission Number: 20888