Go to ICLR 2025 Conference homepageInstruction Following without Instruction Tuning
Adaptations that seem deficient compared to instruction-tuning (finetuning on instruction-response pairs) can still implicitly yield instruction-following language models. We show that training solely on responses, without any corresponding instructions, yields instruction following. This suggests that instruction-tuning just needs to teach the desired distribution of responses. However, we then show that one can finetune on instruction-response pairs from a distribution unlike the test distribution of instructions, like just poetry, or just math, and still yield a model that broadly follows instructions. Instead of acting, e.g., just as math models, these single-task models sometimes behave more as general-purpose chatbots for, e.g., non-math instructions. To begin to explain this implicit instruction tuning, we hypothesize that simple changes to a language model’s distribution yield instruction following. We support this by hand-writing a rule-based adapter that yields instruction-following behavior in language models. The rules are to slowly increase the probability of ending the sequence, penalize repetition, and uniformly change 15 words’ probabilities. In summary, adaptations made without being designed to yield instruction following can do so implicitly.