TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models

ICLR 2026 Conference Submission14632 Authors

Keywords: Knowledge Distillation; Low Rank Adaptation; Fine Tuning

TL;DR: Knowledge distillation for the fine-tuned component of a Large Language Model

Abstract: To embed domain-specific or specialized knowledge into pre-trained foundation models, fine-tuning using techniques such as parameter efficient fine-tuning (e.g. LoRA) is a common practice. However, as new LLM architectures and pre-trained models emerge, transferring this specialized knowledge to newer models becomes an important task. In many scenarios, the original specialized data may be unavailable due to privacy or commercial restrictions, necessitating direct distillation and transfer of this specialized knowledge from the fine-tuned base model to a different pre-trained model. In this work, we present TuneShift-KD, a novel approach that automatically distills specialized knowledge from a fine-tuned model to a target model using only a few examples representative of the specialized information. Our key insight is that specialized knowledge can be identified through perplexity differences between base and fine-tuned models: prompts where the fine-tuned model responds confidently (low perplexity), but the base model struggles (high perplexity), indicate queries corresponding to the specialized knowledge learned by the fine-tuned model. TuneShift-KD leverages this insight to create a synthetic training dataset intended to transfer the specialized knowledge. Using an iterative process, TuneShift-KD generates more prompts that are similar to the prompts that generated responses with specialized knowledge. TuneShift-KD does not require training discriminators or access to training datasets--it is an automated approach that only requires the initial fine-tuned and base models and a few representative prompts. Our experiments demonstrate that models fine-tuned using TuneShift-KD achieve higher accuracy for the fine-tuned specialized knowledge than prior approaches, enabling both ease of deployment and demonstrably more effective transfer of the specialized knowledge.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 14632