GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks
Keywords: In-Context Learning, Example Selection, Large Language Models, Prompting, Prompt Compression, Multitask Training, Compositional Generalization
TL;DR: A novel approach to train example retrievers for in-context learning with state-of-the-art performance on diverse tasks and LLMs.
Abstract: In-context Learning (ICL) is the ability of Large Language Models (LLMs) to perform new tasks when conditioned on prompts comprising a few task examples. However, ICL performance can be critically sensitive to the choice of examples. To dynamically select the best examples for every test input, we propose Example Gisting, a novel approach for training example encoders via supervised fine-tuning with an attention bottleneck between inputs and outputs. These gist models form the basis for GistScore, a novel metric for scoring and selecting informative examples. Further, in addition to fine-tuning gist models on each dataset, we also experiment with training a single model on a large multi-task corpus that can then be used for new tasks out-of-the-box, ensuring a training-free ICL pipeline. Evaluation with 21 datasets spanning 9 tasks and 8 diverse LLMs shows that our fine-tuned models yield state-of-the-art ICL performance with over 20% absolute gain over off-the-shelf retrievers and 5% over the best prior methods. Further, our multi-task model generalizes to new tasks and datasets and is on par or better than all baselines while being three orders faster than the strongest training-free baseline.
Submission Number: 83
Loading