Bayesian-LoRA: Gaussian Process Modeling for Large Language Models

Moule Lin, Shuhao Guan, Andrea Patane, David Gregg, Goetz Botterweck

Keywords: Gaussian Process, LLM, Uncertainty Quantification

Abstract: Pre-trained LLMs are often reasonably calibrated on pre-training like distributions, but fine-tuning them for specific domains often causes a substantial deterioration in calibration, especially on small datasets, leading to overconfident predictions. Although existing Bayesian approaches alleviate this degradation, their computational cost is prohibitive at LLM scale. In this work, we introduce Bayesian-LoRA, which applies a Sparse Gaussian Process (SGP) to the Low-Rank Adaptation (LoRA) fine-tuning approach and integrates a normalizing flow to stabilize the training process, thereby substantially improving calibration in fine-tuned LLMs. We conduct extensive experiments on the LLaMA 2-7B model across a set of commonsense reasoning benchmarks. With only approximately 0.42M additional parameters over LoRA, Bayesian-LoRA reduces the calibration error (ECE) and Negative Log-Likelihood (NLL) without sacrificing accuracy, for both in-distribution and out-of-distribution (OOD) evaluations, while retaining the LoRA’s parameter efficiency and incurring only modest extra training/memory overhead.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 1792