Domain Shift Tuning over Knowledge Gap
Keywords: PEFT, Domain gap, Domain Shift
TL;DR: Domain Shift Tuning (DST), conceptualizes domain gaps as differences in knowledge encapsulated within multiple subnetworks of PLMs and guides the PLM in closing this gap.
Abstract: This paper introduces Domain Shift Tuning (DST), a novel framework designed to guide pre-trained language models (PLMs), including Large Language Models (LLMs), in overcoming domain discrepancies (i.e., source-target).
PLMs, pre-trained on extensive and diverse corpora, the source domain, often encounter domain gaps after fine-tuning over the target domain.
Unlike conventional adapters or Parameter-Efficient Fine-Tuning (PEFT) methods,
DST conceptualizes domain gaps as differences in knowledge encapsulated within multiple subnetworks of PLMs.
To bridge this gap,
our challenge is to find a subnetwork set that corresponds to these pieces of knowledge and their weight.
This direction leads DST to employ a lightweight subnetwork, the Knowledge Steering Layer (KSL), and a training objective, Knowledge Distribution Modeling (KDM).
These components enable DST to fine-tune PLMs by aligning the knowledge weights of the source domain with those of the target domain.
Experimental results on diverse datasets demonstrate that DST effectively mitigates the domain gap, allowing PLMs to generate text that closely aligns with even a small target corpus, thereby significantly enhancing domain adaptation for PLMs at lower computational cost.
Primary Area: optimization
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5324
Loading