Consistent Low-Rank Adaptation of Two-layer Neural Networks: A Nonparametric Statistics Approach

ICLR 2026 Conference Submission19072 Authors

Keywords: Low-rank model, Transfer Learning, Deep Neural Networks, Stein's Lemma, Ridge regression

Abstract: Low-Rank Adaptation (LoRA) is a powerful technique for fine-tuning Large Language Models (LLMs), offering greater parameter efficiency and improved generalization in data-constrained settings. While its advantages makes it highly promising for general transfer learning, its reliance on iterative optimization methods such as SGD still demands substantial computation and poses a challenge for theoretical analysis.We propose a novel two-step, closed-form approach for LoRA in two-layer feedforward neural networks (FNN) that mitigates the reliance on iterative algorithms. First, by leveraging Stein’s lemma, a classical statistical tool, we derive an analytical estimator for the first-layer LoRA parameters. Second, we solve for the second-layer parameters via reduced-rank ridge regression. We provide theoretical guarantees for the low-rank parameter estimation under a projection adaptation assumption: the optimal first layer adaptation removes irrelevant directions via subspace projection. This generalizes the concept of rank pruning, which removes irrelevant low-rank components from a weight matrix.Crucially, our solution is non-iterative and computationally efficient, computing the full adaptation in seconds—a fraction of the time required by SGD-based LoRA. Numerical experiments on MNIST suggest that our method not only significantly reduces computational cost and achieves prediction performance comparable to that of a fully trained LoRA model, but also serves as a good initialization for SGD-based LoRA.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 19072