Latent Space Factorization in LoRA

Shashi Kumar, Yacouba Kaloga, John Mtr., Petr Motlicek, Ina Kodrasi

Keywords: low-rank adaptation, latent space factorization, spurious correlation robustness, fvae-lora

Abstract:

Low-rank adaptation (LoRA) is a widely used method for parameter-efficient finetuning.
However, existing LoRA variants lack mechanisms to explicitly disambiguate task-relevant information within the learned low-rank subspace, potentially limiting downstream performance.
We propose Factorized Variational Autoencoder LoRA (FVAE-LoRA), which leverages a VAE to learn two distinct latent spaces.
Our novel Evidence Lower Bound formulation explicitly promotes factorization between the latent spaces, dedicating one latent space to task-salient features and the other to residual information.
Extensive experiments on text, audio, and image tasks demonstrate that FVAE-LoRA consistently outperforms standard LoRA.
Moreover, spurious correlation evaluations confirm that FVAE-LoRA better isolates task-relevant signals, leading to improved robustness under distribution shifts.
Our code is publicly available at: https://github.com/idiap/FVAE-LoRA

Primary Area: Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Submission Number: 17783