FT++: Revitalizing Fine-Tuning for Robust Knowledge Editing

ACL ARR 2026 January Submission3609 Authors

Keywords: Knowledge Editing, Large Language Models, Fine-Tuning, Catastrophic Forgetting, Regularization

Abstract: Knowledge editing aims to correct outdated or erroneous information in Large Language Models (LLMs) without degrading their general capabilities. Recent approaches have largely moved away from standard Fine-Tuning (FT), citing its susceptibility to catastrophic forgetting, and instead favor complex, localized architectural modifications. In this work, we challenge this consensus. We demonstrate that the limitations of FT are not inherent but stem from unconstrained optimization. We introduce FT++, an enhanced fine-tuning framework that integrates three strategic regularizations: label smoothing to prevent overfitting, a general knowledge loss to preserve global distribution, and a novel relation-aware local loss to maintain semantic stability. Extensive experiments on ZsRE and COUNTERFACT show that FT++ significantly outperforms state-of-the-art methods (including ROME and MEMIT) in both single and batch editing scenarios. Our findings establish that properly regularized fine-tuning is not merely a baseline, but a superior, robust, and efficient solution for knowledge editing.

Paper Type: Long

Research Area: Language Models

Research Area Keywords: fine-tuning, continual learning, robustness

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 3609