CollabMask: Neuron Collaboration Gradient Masks for LLM Fine-Tuning

ACL ARR 2026 January Submission9053 Authors

Keywords: LLM, gradient mask, neuron level explainability, neuron collaboratin, finetuning, continual learning, catastrophic forgetting

Abstract:

The rapid advancement of large language models (LLMs) has increased the need for effective task-specific adaptation. Fine-tuning remains the primary approach but often suffers from redundant parameter updates and catastrophic forgetting in continual learning settings. Existing methods mitigate these issues using gradient masks, yet they largely ignore interactions among neurons. We observe neuron collaboration, where neurons tend to have closely connected neighbors that are co-activated for specific tasks. Leveraging this concept, we propose CollabMask (Collaborative Neuron Mask Fine-tuning), which generates dynamic, collaboration-aware gradient masks to induce gradient sparsity.
Experiments on mathematical tasks show a 2.7\% improvement over full-parameter fine-tuning and a 3.9\% improvement over other gradient masking methods. In continual learning settings, CollabMask achieves performance within 1\% of the best baseline on new tasks, previously learned tasks, and general reasoning ability, demonstrating its effectiveness in enhancing fine-tuning and mitigating catastrophic forgetting. We also discuss the connection between neuron collaboration and neuron-level interpretability, highlighting its potential to improve the explainability of LLMs.

Paper Type: Long

Research Area: Special Theme (conference specific)

Research Area Keywords: Language Modeling, Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 9053