ManifoldLoRA: Geometry-Aware Adaptive Rank Selection via Circuit Manifold Probes

Keywords: parameter-efficient fine-tuning, LoRA, adaptive rank selection, representation geometry, circuit probes, transformer interpretability, manifold alignment

TL;DR: ManifoldLoRA combines adaptive LoRA rank gating with circuit manifold probes to align updates with representation geometry.

Abstract:

Low-rank adaptation (LoRA) has become the dominant paradigm for
parameter-efficient fine-tuning of large language models, yet existing
methods fix the adapter rank prior to training and ignore the intrinsic
geometric structure of the representations being adapted.
We introduce \textbf{ManifoldLoRA}, a method that couples adaptive,
gate-controlled rank selection with \emph{circuit manifold probes}—
lightweight orthonormal bases that track the activation geometry of
each transformer layer throughout training.
A novel manifold alignment loss encourages every LoRA weight update to
lie within the learned low-dimensional subspace of its layer's
representations, preventing off-manifold drift and improving geometric
coherence.
Experiments fine-tuning Qwen3-14B on a stratified mixture of
instruction-following and preference data demonstrate that
\textit{(i)} query projections exploit significantly higher effective
rank than value projections, revealing a natural asymmetry in attention
geometry; \textit{(ii)} manifold alignment loss converges from unity to
near zero over training, confirming progressive subspace alignment; and
\textit{(iii)} participation ratio analysis exposes the intrinsic
dimensionality profile of residual-stream representations across all
forty layers.
ManifoldLoRA provides both a practical fine-tuning algorithm and a
diagnostic framework for understanding the low-rank geometry of
large-scale language models.

Submission Number: 131