A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Search

Baek Seong-Eun, Lee Jung-Mok, Kim Sung-Bin, Tae-Hyun Oh

Keywords: Bayesian optimization, LoRA hyperparameter optimization

Abstract: Fine-tuning Large Language Models (LLMs) with Low-Rank Adaptation (LoRA) is resource-efficient, but its performance is highly sensitive to hyperparameter choices, making exhaustive search expensive. To address this, we propose a framework that integrates pre-trained LLM knowledge into Bayesian Optimization (BO) for efficient LoRA hyperparameter optimization. Our method uses an LLM as a discrete-to-continuous mapping module that converts hyperparameter configurations and domain-aware prompts into continuous embeddings, where BO is performed. The prompts describe the roles and relationships of LoRA hyperparameters, while an additional learnable token captures information not easily expressed in text. We further introduce proxy evaluation on a data subset, exploiting its strong correlation with full-data training to reduce evaluation cost. Experiments show that our method finds strong hyperparameters within about 30 iterations, achieving over 20% improvement over standard hyperparameters selected from roughly 45,000 combinations.

Submission Number: 89