IA2: Alignment with ICL Activations improves Supervised Fine-Tuning

Aayush Mishra, Daniel Khashabi, Anqi Liu

Keywords: In Context Learning, ICL, Supervised Fine Tuning, SFT, Adaptation, Self-Distillation, Distillation

TL;DR: Use activations produced during ICL to align SFT models' functional behavior with ICL. This results in better accuracy and calibration in SFT models.

Abstract:

Supervised Fine-Tuning (SFT) is used to specialize model behavior by training weights to produce intended target responses for queries. In contrast, In-Context Learning (ICL) adapts models during inference with instructions or demonstrations in the prompt.
ICL can offer better generalizability and more calibrated responses compared to SFT in data scarce settings, at the cost of more inference compute. In this work, we ask the question: \textit{Can ICL's internal computations be used to improve the qualities of SFT?} We first show that ICL and SFT produce distinct activation patterns, indicating that the two methods achieve adaptation through different functional mechanisms. Motivated by this observation and to use ICL's rich functionality, we introduce \textbf{I}CL \textbf{A}ctivation \textbf{A}lignment (\act), a self-distillation technique which aims to replicate ICL's activation patterns in SFT models and incentivizes ICL-like internal reasoning. Performing \act as a priming step before SFT significantly improves the accuracy and calibration of model outputs, as shown by our extensive empirical results on 12 popular benchmarks and two model families. This finding is not only practically useful, but also offers a conceptual window into the inner mechanics of model adaptation.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 3556