Demonstration Distillation for Efficient In-Context Learning

Tong Chen, Qirun Dai, Zhijie Deng, Dequan Wang

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: demonstration, distillation, in-context learning, large language model, llm

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: In-context learning (ICL) substantially amplifies the predictive capability of large language models (LLMs), where the prompt typically contains a few question-answer pairs termed demonstrations, and a final question. Although lengthy and information-rich demonstrations can improve performance, they also inflate the computational burdens and financial costs, sometimes even breaching the context limit of LLMs. Existing solutions, such as prompt selection or context compression, frequently neglect the presence of superfluous information within these elongated prompts. To bridge the gap, this paper introduces demonstration distillation, a novel paradigm that targets excising the redundant content in the prompt without sacrificing ICL efficacy. We propose a distillation framework, Distillist-Generalist-Specialist (DGS), as an automated solution without additional model training. DGS iteratively refines the demonstration with the aid of three LLM-powered agents, eliminating superfluous information while maintaining valuable knowledge. Evaluations on three diverse datasets—GSM8K, BoolQ, and MultiRC—reveal the robustness and effectiveness of DGS. Particularly, DGS realizes $1.5-2$, $3-6$, and $1.5-3$ distillation ratios without compromising ICL performance on the three datasets.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6763