GenKnowSub: Improving Modularity and Reusability of LLMs through General Knowledge Subtraction

ACL ARR 2025 February Submission4468 Authors

Abstract:

Large language models (LLMs) often struggle with zero-shot generalization, and several modular approaches have been proposed to address this challenge. Yet, we hypothesize that a key limitation remains: the entanglement of general knowledge and task-specific adaptations. To overcome this, we propose a modular framework that disentangles these components by constructing a library of task-specific LoRA modules alongside a general-domain LoRA. By subtracting this general knowledge component from each task-specific module, we obtain residual modules that focus more exclusively on task-relevant information. We call this approach general knowledge subtraction or GenKnowSub. Leveraging the refined task-specific modules and the Arrow routing algorithm, we dynamically select and combine modules for new inputs without additional training.
Our studies on the Phi-3 model and standard Arrow as baselines reveal that using general knowledge LoRAs derived from diverse languages, including English, French, and German, yields consistent performance gains in both monolingual and cross-lingual settings across a wide set of benchmarks.
Further experiments on Phi-2 reveal how GenKnowSub generalizes to a weaker LLM.

Paper Type: Short

Research Area: Special Theme (conference specific)

Research Area Keywords: Generalisation of NLP Models, Zero-Shot Transfer Learning, Modular Deep Leaning

Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency

Languages Studied: English, French, German

Submission Number: 4468