TaiLoR: Protecting Principal Components in Parameter-Efficient Continual Learning

ACL ARR 2026 May Submission15061 Authors

Keywords: continual learning, low-rank adaptation, PEFT, SVD

Abstract: Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in Continual Learning. In this paper we introduce TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.

Paper Type: Short

Research Area: Machine Learning for NLP

Research Area Keywords: Continual Learning, PEFT, spectral

Languages Studied: English, German, code

EMNLP 2026 AI Reviewing Experiment: yes

Submission Number: 15061