Forget-to-Focus: Can unlearning Improve Domain Specialization in LLMs?

Chaitali Bhattacharyya, Yeseong Kim

Keywords: llm unlearning

Abstract: Standard fine-tuning of Large Language Models for domain-specific tasks is often suboptimal due to interference from vast, pre-existing general knowledge from pretraining, leading to issues like negative knowledge transfer and the reinforcement of spurious correlations. We study whether removing parts of a pretrained model’s pre-existing general knowledge before adaptation can make downstream learning easier. We propose and analyze Forget-to-Focus: a two-stage protocol that first performs targeted unlearning on a “forget” set (with an optional retain set for stability, then fine-tunes on a domain-specific dataset. Through rigorous experiments on different domains such as medical, mathematics, and coding benchmarks, we analyze whether this preparatory unlearning can lead to improved domain specialization. Our findings show that this protocol consistently outperforms standard fine-tuning e.g., it improves HumanEval pass@1 by 32.5\% on Qwen3-0.6B and 11.95\% on Qwen 72B model compared to standard fine-tuning. Beyond accuracy, we observe that F2F reshapes representational geometry as measured by centered kernel alignment, shifting models away from generalist initialization toward structures more conducive to in-domain specialization. Furthermore, unlearning prior fine-tuning helps improved calibration on medical QA tasks, reducing overconfidence and mitigating reliability issues that persist under standard fine-tuning. These findings establish unlearning not merely as a privacy tool but as a principled intervention for domain adaptation. By strategically suppressing irrelevant pretraining knowledge, Forget-to-Focus helps more stable optimization dynamics, better calibrated predictions, and consistently stronger downstream performance. The code is available at anonymous github : \href{https://anonymous.4open.science/r/D-1545/README.md}{https://anonymous.4open.science/r/D-1545/README.md}

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 2718