Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapter

Haotian Xiang, Bingcong Li, Qin Lu

Keywords: Uncertainty Quantification, Bayesian Neural Network, Bayesian last layer, Large Language Models, Parameter-Efficient Fine-Tuning, Orthogonal Parametrization

TL;DR: PoLAR-VBLL combines orthogonalized low-rank adapters with variational Bayesian inference on the last layer to achieve scalable, well-calibrated uncertainty quantification for fine-tuned LLMs while maintaining high accuracy.

Abstract:

When deploying large language models (LLMs) to safety-critical applications, uncertainty quantification (UQ) is of utmost importance to self-assess the reliability of the LLM-based decisions. However, such decisions typically suffer from overconfidence, particularly after parameter-efficient fine-tuning (PEFT) for downstream domain-specific tasks with limited data.
To address these limitations, we build on the Bayesian last layer (BLL) model, where the LLM-based ${\it deterministic}$ feature extractor is followed by random LL parameters for uncertainty reasoning.
Since existing low-rank adapters (LoRA) for PEFT have limited expressiveness due to rank collapse, we address this with Polar-decomposed Low-rank Adapter Representation (PoLAR), an orthogonalized parameterization paired with Riemannian optimization to enable more stable and expressive adaptation.
The resulting PoLAR-VBLL is a flexible framework that nicely integrates architecture-enhanced optimization with scalable Bayesian inference to endow LLMs with well-calibrated UQ.
Our empirical results verify the effectiveness of PoLAR-VBLL in terms of generalization and uncertainty estimation on both in-distribution and out-of-distribution data for various common-sense reasoning tasks.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 13666