Efficient Prompting via Dynamic In-Context Learning

TMLR Paper4738 Authors

Abstract: In context learning has become a common practice for prompting generalist models. Despite being effective, in-context learning can be computationally inefficient because it makes the input prompt much longer, consuming valuable space in the context window and leading to larger computational costs. In this paper, we propose DynaICL, a recipe for efficient prompting with black-box generalist models that dynamically allocates in-context examples according to the input complexity and the computational budget. We train a meta controller that predicts the number of in-context examples suitable for the generalist model to make a good prediction based on the difficulty of a specific input. We then dynamically allocate the number of demonstrations for an input according to the computation budget. Experimental results show that DynaICL helps achieve a better performance-efficiency trade-off in two practical settings where we have constraints on computational resources or the minimum required performance. Specifically, DynaICL saves up to 46% token budget compared to the common practice that allocates the same number of in-context examples to each input. In addition, we also find that a meta controller trained on a certain backbone model and tasks can successfully generalize to unseen models and tasks, suggesting that we can train a meta controller once and use it in various use cases.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Jonathan_Berant1

Submission Number: 4738