Variational Low-Rank Adaptation Using IVON

Bai Cong, Nico Daheim, Yuesong Shen, Daniel Cremers, Rio Yokota, Mohammad Emtiyaz Khan, Thomas Möllenhoff

Keywords: Large language model, Low-rank adaption, Variational learning, Model calibration

TL;DR: We use the IVON optimizer to improve the accuracy and calibration in LoRA finetuning for large language models.

Abstract: We use variational learning to improve accuracy and calibration of low-rank adaptation (LoRA) finetuning in large language models. Specifically, we employ the Improved Variational Online Newton (IVON) optimizer, which is a drop-in replacement of AdamW but significantly improves the performance with negligible overhead. We test our method by finetuning a Llama 2 model with 7 billion parameters on a range of commonsense reasoning datasets. Compared to AdamW finetuning, IVON improves accuracy by 2.8% and ECE by 4.6% on average. Our work provides more evidence for the effectiveness of variational learning in large language models. A link to the code will be provided in the final paper.

Submission Number: 56