Learning Adapter Rank via Symmetry Breaking

Cooper Doyle, Andy Hu, Rebecca Chan, Anna Leontjeva

Keywords: variational dropout, automatic relevance determination, low-rank adaptation, uncertainty quantification, gauge symmetry breaking

TL;DR: BayesLoRA jointly learns effective adapter rank and predictive uncertainty with only $\mathcal{O}(r)$ additional parameters by breaking LoRA gauge symmetry

Abstract: Low-rank adaptation is effective partly because downstream updates lie in a low-dimensional subspace, but the latent rank coordinates of LoRA are not identifiable: any invertible reparameterization of the adapter factors leaves the weight update unchanged. We show that a rank-wise variational-dropout posterior turns this non-identifiability into a useful inductive bias. By breaking LoRA’s rotational gauge symmetry, the variational objective selects a preferred basis in rank space, enabling automatic relevance determination over rank directions. This yields Low-Rank Variational Dropout (LRVD), a framework for uncertainty and sparsification directly in the low-rank adaptation space rather than the ambient weight space. As an instantiation, BayesLoRA jointly learns effective adapter rank and rank-space predictive uncertainty with only O(r) additional parameters. Empirically, BayesLoRA induces stable rank structure aligned with dominant singular directions of learned updates, matches or exceeds strong low-rank sparsification baselines at comparable training cost, and exposes a compact accuracy–compression–calibration trade-off relative to richer Bayesian LoRA methods.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 83