Learning to Reason with Autoregressive In-Context Distillation
Keywords: Knowledge Distillation, In-Context Learning, Reasoning, Large Language Models
TL;DR: We present AICD, a novel learning paradigm that jointly distills in-context learning and reasoning capability from LLM to smaller LMs.
Abstract: We investigate the joint distillation of in-context learning and reasoning from advanced large language models (LLMs) to their smaller counterparts. We introduce Autoregressive In-Context Distillation (AICD), a simple yet effective paradigm for this purpose. AICD employs meta-teacher forcing on chain-of-thought (CoT) examples and leverages the autoregressive nature of LLMs to jointly optimize the likelihood of all rationales in-context. Experiments on both mathematical and commonsense reasoning tasks demonstrate the efficacy of AICD. Furthermore, AICD enhances the capability of student LLMs in generating meaningful CoTs.
Submission Number: 18
Loading