Can language models learn from explanations in context?

Andrew Kyle Lampinen, Ishita Dasgupta, Stephanie C.Y. Chan, Kory Wallace Mathewson, Michael Henry Tessler, Antonia Creswell, James McClelland, Jane X Wang, Felix Hill

Abstract:

Language Models (LMs) can perform new
tasks by adapting to a few in-context examples.
For humans, explanations that connect examples to task principles can improve learning.
We therefore investigate whether explanations
of few-shot examples can help LMs. We annotate questions from 40 challenging tasks with
answer explanations, and various matched control explanations. We evaluate how different
types of explanations, instructions, and controls affect zero- and few-shot performance.
We analyze these results using statistical multilevel modeling techniques that account for the
nested dependencies among conditions, tasks,
prompts, and models. We find that explanations can improve performance—even without tuning. Furthermore, explanations handtuned for performance on a small validation set
offer substantially larger benefits, and building a prompt by selecting examples and explanations together substantially improves performance over selecting examples alone. Finally, even untuned explanations outperform
carefully matched controls, suggesting that the
benefits are due to the link between an example and its explanation, rather than lower-level
features. However, only large models benefit.
In summary, explanations can support the incontext lea