ProDA: Profile-Decomposed Adaptation for Capturing the Structured Residual in Low-Rank Updates

Yupeng Chang, Yuan Wu, Yi Chang

Keywords: parameter-efficient fine-tuning, large language models, low-rank adaptation, ProDA, weight decomposition, synergistic adaptation, structured residual

TL;DR: We identify a structured "missing profile" in LoRA's low-rank updates and propose ProDA, a new method that synergistically models this profile to significantly outperform existing PEFT methods.

Abstract: Low-Rank Adaptation (LoRA), a cornerstone of parameter-efficient fine-tuning (PEFT), relies on the core assumption that weight updates are inherently low-rank. We challenge this premise, revealing that this low-rank approximation systematically neglects a significant and highly structured residual, which we term the **missing profile**. To harness this insight, we introduce Profile-Decomposed Adaptation (ProDA), a novel method that captures this residual using highly efficient, axis-aligned vectors. Critically, instead of treating this component as a static error to be corrected, ProDA integrates it multiplicatively, allowing it to dynamically re-scale and modulate the primary low-rank update. Our extensive experiments validate the effectiveness of this approach. ProDA establishes a new state-of-the-art on commonsense reasoning benchmarks and, remarkably, surpasses even full fine-tuning on the GLUE benchmark, suggesting it can act as a powerful regularizer that fosters generalizability. Moreover, on complex generative tasks where standard LoRA falters, ProDA dramatically narrows the performance gap to full fine-tuning. These findings validate our central thesis: the structured residual in PEFT is not mere noise, but a rich signal for synergistic exploitation.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 4272