NoLoRA: Nonlinear Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

ICLR 2026 Conference Submission9597 Authors

Keywords: Low-Rank Adaptation, Parameter-Efficient Fine-Tuning, Nonlinear

Abstract: Low-Rank Adaptation (LoRA) has been widely adopted for parameter-efficient fine-tuning of large language models, as it enables effective adaptation while maintaining efficiency. However, existing LoRA methods are fundamentally linear in nature, relying on the multiplication of two matrices (\textit{B×A}) for parameter adaptation. This inherently linear structure constrains their expressiveness, rendering them insufficient for capturing higher-order feature interactions and complex nonlinear patterns essential for advanced tasks. Consequently, this linearity becomes a bottleneck that limits further performance improvements. To address this limitation, we propose a nonlinear extension that introduces a learnable nonlinearity and a modulation mechanism into the low-rank adapter (Nonlinear Low-Rank Adaptation, NoLoRA), enhancing adaptability across diverse tasks. Our design preserves the parameter efficiency and scalability of LoRA while significantly improving representational capacity. Comprehensive experiments on four benchmarks, including commonsense reasoning, natural language understanding, image classification, and mathematical reasoning, demonstrate that our approach achieves consistent and substantial improvements over vanilla LoRA, LoRA's variant and other Parameter-efficient fine-tuning (PEFT) methods, with negligible additional computational overhead. These findings suggest that incorporating lightweight nonlinear structures into parameter-efficient fine-tuning frameworks offers a promising direction for improving the adaptability of large models.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 9597