Parameter-Efficient Subspace Optimization for LLM Fine-Tuning

ICLR 2026 Conference Submission21660 Authors

Keywords: parameter-efficient fine tuning, subspace minimization, intrinsic dimension, large language model, LoRA

Abstract: This paper develops a new perspective on parameter-efficient training for LLMs, inspired by the classical theory of subspace minimization. We introduce a unifying framework, Parameter-Efficient Subspace Optimization (PESO), which not only recovers many existing methods such as LoRA but also bridges them with the principled algorithmic and theoretical foundations of subspace optimization. This connection highlights a natural ``exploration--exploitation'' view of subspace methods, guiding the design of new algorithms that achieve strong convergence performance while still preserving memory efficiency. Importantly, our framework establishes the first convergence in the full-parameter space, resolving a critical gap in the current literature where low-rank updates lack such guarantees. We further instantiate the framework into a practical algorithm named PESO-LoRA, based on LoRA-type parameterization. Our algorithm achieves notable improvements over existing methods on standard benchmarks.

Primary Area: optimization

Submission Number: 21660