Outlier-Free Genomic Foundation Models for Resource-Efficient Training and Low-Bit Inference

Chenghao Qiu, Haozheng Luo, Maojiang Su, Zhihan Zhou, Zoe Mehta, Guo Ye, Jerry Yao-Chieh Hu, Han Liu

Keywords: Genomic Foundation Models, Outlier Removal, Low-Bit Quantization, LoRA, Resource-Constrained Settings

TL;DR: GERM introduces an outlier-free genomic foundation model with efficient LoRA and 4-bit quantization, accelerating training and inference on resource-constrained devices compared to sota methods.

Abstract: While genomic foundation models (GFMs) hold significant potential for biological discovery, their large parameter sizes and high computational demands imit practical deployment on resource-constrained devices. We propose GERM, an outlier-free architecture that replaces standard attention with an outlier-free mechanism, achieving both accelerated low-rank adaptation and robust post-training quantization. enhances both training and inference via outlier removal. We further propose GERM-T, a small-step continual learning strategy with outlier-free framework that leverages existing checkpoints to avoid costly retraining from scratch. Our experiments demonstrate GERM’s superiority over state-of-the-art GFMs: it achieves 37.98% higher fine-tuning performance and improves quantization performance by 64.34% , alongside 92.14% reduction in average kurtosis and 82.77% lower maximum infinity norm. Notably, GERM enables rapid deployment on edge devices, completing DNABERT-2 fine-tuning in 5 minutes on a single 2080Ti GPU with 34.9% faster training, 24.79% inference acceleration, and robust 4-bit quantization. GERM consistently delivers superior performance, making it a practical solution for deploying GFMs in resource-constrained settings.

Submission Number: 18