Fast and Low-Cost Genomic Foundation Models via Outlier Removal

Haozheng Luo, Chenghao Qiu, Maojiang Su, Zhihan Zhou, Zoe Mehta, Guo Ye, Jerry Yao-Chieh Hu, Han Liu

TL;DR: GERM is a genomic foundation model optimized for low-resource settings by removing outliers, enhancing low-rank adaptation and quantization, achieving up to 64.34% efficiency gains and 37.98% better fine-tuning performance over baseline models.

Abstract: To address the challenge of scarce computational resources in genomic modeling, we introduce GERM, a genomic foundation model optimized for accessibility and adaptability. GERM improves upon models like DNABERT-2 by eliminating outliers that hinder low-rank adaptation and post-training quantization, enhancing both efficiency and robustness. We replace the vanilla attention layer with an outlier-free mechanism inspired by associative memory models. By removing outliers during both pre-training and fine-tuning, this approach accelerates adaptation, reduces computational costs, and enhances quantization robustness within acceptable loss margins. Additionally, we propose GERM-T, a strategy that employs small-step continual learning within the outlier-free framework, leveraging original checkpoints to avoid retraining from scratch. Empirically, GERM improves fine-tuning performance by 37.98% and quantization by 64.34% over the baseline model. It also reduces average kurtosis by 92.14% and maximum infinity norm by 82.77%. Compared to leading methods, GERM consistently delivers superior performance, offering a practical solution for genomic modeling in resource-constrained settings.

Lay Summary:

Large AI models for genomic analysis often require significant computing resources, making them difficult to use in resource-constrained environments. To address this, we introduce GERM, a genomic foundation model optimized for accessibility and adaptability. GERM improves upon models like DNABERT-2 by removing outliers that impede low-rank adaptation and post-training quantization, thereby improving efficiency and robustness. It replaces the standard attention layer with an outlier-free mechanism inspired by associative memory models, enabling faster adaptation, lower computational cost, and improved quantization robustness with minimal performance loss. We also propose GERM-T, a continual learning strategy that supports small-step updates using existing checkpoints, avoiding the need for full retraining.

Our paper reveals that GERM improves fine-tuning performance by 37.98% and quantization by 64.34% over the baseline. Moreover, it effectively suppresses outlier indicators, achieving a 92.14% reduction in average kurtosis and an 82.77% decrease in the maximum infinity norm. These results make GERM and GERM-T practical tools for genomic modeling in resource-constrained settings.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Link To Code: https://github.com/MAGICS-LAB/GERM

Primary Area: Applications->Everything Else

Keywords: DNA, Outlier-free, Genomic Foundation Models, Outlier Removal

Submission Number: 832