RAPA: Recursively Aligned Pathway Adaptation of Large Language Models

ICLR 2026 Conference Submission13453 Authors

Keywords: large language model, parameter-efficient fine-tuning, index-aligned adaptation, structured adaptation

TL;DR: We propose Pathway-Aligned Tuning (PAT), an adapter-free PEFT that builds cross-layer pathways by tuning balanced square submatrices and aligning indices. Across most benchmarks, PAT outperforms leading baselines while preserving efficiency.

Abstract: Parameter-Efficient Fine-Tuning (PEFT) adapts large language models (LLMs) by training only a small fraction of parameters. Adapter-based approaches reduce compute per step but introduce practical overhead from the additional adapter path (e.g., extra kernel launches and activation storage). Adapter-free approaches avoid this structural overhead by directly updating pretrained weights; however, per-layer random index selection can fragment the trainable subspace, attenuating gradient flow and limiting accuracy. We propose **Recursively Aligned Pathway Adaptation (RAPA)**, an adapter-free PEFT method that forms index-consistent pathways through depth. RAPA follows two principles: (i) selecting balanced submatrices that maximize the number of weights alignable across layers, and (ii) recursively aligning these indices across layers and residual connections. In experiments, RAPA matches or surpasses strong PEFT baselines across most benchmarks while preserving adapter-free efficiency with minimal memory and compute overhead. Code is available at \url{https://anonymous.4open.science/r/rapa}.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 13453