Locate-then-Unlearn: An Effective Method of Multi-Task Continuous Learning for Large Language Models

Songshi Liang, Hongda Sun, Ting-En Lin, Yuchuan Wu, Zihe Wang, Yongbin Li, Rui Yan

Keywords: Machine unlearning, Continue learning, Model editing

Abstract:

Nowadays large language models (LLMs) have achieved remarkable success in
various NLP tasks. However, they often misinterpret human instructions and generate incorrect or outdated responses, highlighting the need for more effective continual learning techniques. While recent efforts have introduced unlearning methods to remove erroneous knowledge, existing approaches still struggle in multi-task learning scenarios. To overcome these limitations, we propose Locate-then-unlearn, a new framework that identifies and selectively unlearns task-specific
neurons to enable efficient multi-task learning. We hypothesize that LLM neurons can be broadly categorized into task-specific neurons for handling individual
tasks, and general neurons to maintain the model’s foundational capabilities. To
accurately identify task-specific neurons, the locating process includes: (1) ranking task-related neurons based on their importance to each task, and (2) identifying
task-specific neurons by applying intervention to assess how neuron activity impacts task performance, isolating those most critical to each task. We conduct
comprehensive evaluations in two experimental setups: single-task specialization
and multi-task generalization. The results show that our method significantly improves performance across both settings. This indicates that our method effectively balances model efficiency and accuracy in multi-task continual learning.

Primary Area: interpretability and explainable AI

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3322