SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA

ACL ARR 2025 July Submission1471 Authors

Abstract: Parameter-Efficient Fine-Tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA), are indispensable for efficiently customizing Large Language Models (LLMs). However, vanilla LoRA suffers from slow convergence speed and knowledge forgetting problems. Recent studies have leveraged the power of designed LoRA initialization, to enhance the fine-tuning efficiency, or to preserve knowledge in the pre-trained LLM. However, none of these works can address the two cases at the same time. To this end, we introduce \textbf{S}ubspace-\textbf{C}onstrained LoRA (\textbf{SC-LoRA}), a novel LoRA initialization framework engineered to navigate the trade-off between efficient fine-tuning and knowledge preservation. We achieve this by constraining the output of trainable LoRA adapters in a low-rank subspace, where the context information of fine-tuning data is most preserved while the context information of preserved knowledge is least retained, in a balanced way. Such constraint enables the trainable weights to primarily focus on the main features of fine-tuning data while avoiding damaging the preserved knowledge features. We provide theoretical analysis on our method, and conduct extensive experiments including \textit{safety preservation} and \textit{world knowledge preservation}, on various downstream tasks. In our experiments, SC-LoRA succeeds in delivering superior fine-tuning performance while markedly diminishing knowledge forgetting, surpassing contemporary LoRA initialization methods.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: fine-tuning,safety and alignment

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Theory

Languages Studied: English

Previous URL: https://openreview.net/forum?id=zDYbAi8Ub7

Reassignment Request Area Chair: Yes, I want a different area chair for our submission

Reassignment Request Reviewers: Yes, I want a different set of reviewers

Justification For Not Keeping Action Editor Or Reviewers: Requesting reassignment due to recurring misalignment with the paper's scope: (1) misconception that the experimental design suffered from test data leakage; (2) dismissal of standard LLM-evaluation protocols without proposing alternatives; (3) insistence on inapplicable baseline comparisons despite methodological clarifications. These patterns hindered consensus-building on revision priorities.

A1 Limitations Section: This paper has a limitations section.

A2 Potential Risks: N/A

A2 Elaboration: We do not create or release harmful datasets or models, or jailbreaking algorithms.

B Use Or Create Scientific Artifacts: Yes

B1 Cite Creators Of Artifacts: Yes

B1 Elaboration: section 4

B2 Discuss The License For Artifacts: N/A

B3 Artifact Use Consistent With Intended Use: Yes

B3 Elaboration: section 4

B4 Data Contains Personally Identifying Info Or Offensive Content: No

B4 Elaboration: We do not create or release harmful datasets or models, or jailbreaking algorithms.

B5 Documentation Of Artifacts: N/A

B5 Elaboration: We do not release such artifacts.

B6 Statistics For Data: Yes

B6 Elaboration: section 4

C Computational Experiments: Yes

C1 Model Size And Budget: Yes

C1 Elaboration: section4

C2 Experimental Setup And Hyperparameters: Yes

C2 Elaboration: section 4

C3 Descriptive Statistics: Yes

C3 Elaboration: section 4

C4 Parameters For Packages: Yes

C4 Elaboration: section 4

D Human Subjects Including Annotators: No

D1 Instructions Given To Participants: N/A

D2 Recruitment And Payment: N/A

D3 Data Consent: N/A

D4 Ethics Review Board Approval: N/A

D5 Characteristics Of Annotators: N/A

E Ai Assistants In Research Or Writing: Yes

E1 Information About Use Of Ai Assistants: No

E1 Elaboration: The use of AI assistants is very slight. We only use AI assistants to correct our grammar.

Author Submission Checklist: yes

Submission Number: 1471