Astra: Activation-Space Tail Eigenvector Low-Rank Adaptation of Large Language Models

ICLR 2026 Conference Submission19165 Authors

Keywords: Parameter Efficient Fine-Tuning (PEFT); Large Language Models (LLMs); Low-Resource/Efficient Methods for ML

Abstract: Parameter-Efficient Fine-Tuning (PEFT) methods, especially LoRA, are widely used for adapting pre-trained models to downstream tasks due to their computational and storage efficiency. However, LoRA and its successors often focus on well-optimized principal subspaces of model activations, yielding diminishing returns and potentially destabilizing pretrained representations, while the subspaces correspond to tail eigenvectors remain largely under-utilized. In this work, we propose Astra (Activation-Space Tail-Eigenvector Low-Rank Adaptation), a novel PEFT method that leverages the tail eigenvectors of the model output activations—estimated from a small task-specific calibration set—to construct task-adaptive low-rank adapters. By constraining updates to the subspace spanned by the tail eigenvectors of output activations, Astra avoids interfering with pretrained task-relevant semantic structure and adapts in directions that minimize energy in the original task-specific representational space, leading to faster convergence and improved downstream performance. Extensive experiments across natural language understanding (NLU) and natural language generation (NLG) tasks demonstrate that Astra consistently outperforms existing PEFT baselines across 16 benchmarks and even surpasses full fine-tuning (FFT) in certain scenarios.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 19165