BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation for Overfitting-Resilient Fine-Tuning of Biological Foundation Models

Peijia Qin, Ruiyi Zhang, Pengtao Xie

Keywords: Large Language Models, Protein Language Models, Parameter-Efficient Fine-Tuning

TL;DR: We propose BiDoRA, a bi-level optimization based PEFT method that decouples magnitude and direction updates for reducing overfitting in fine-tuning biological foundation models.

Abstract:

Biological foundation models (e.g., protein language models) are typically fine-tuned on small and noisy datasets, making overfitting a central challenge. We present BiDoRA, an overfitting-resilient parameter-efficient fine-tuning (PEFT) method tailored for foundation models.
BiDoRA builds on weight-decomposed low-rank adaptation (DoRA) but addresses its over-expressiveness by decoupling magnitude and direction optimization within a bi-level optimization (BLO) framework: the direction is learned on a training split with magnitudes fixed, while magnitudes are updated on a validation split via hypergradient descent. This design reduces overfitting and yields update patterns that better mimic full fine-tuning under the same parameter budget. On a broad suite of biological and natural language tasks, BiDoRA matches or surpasses strong PEFT baselines. Code is available at https://github.com/t2ance/BiDoRA

Submission Number: 33