Why is prompting hard? Understanding prompts on binary sequence predictors

Li Kevin Wenliang, Anian Ruoss, Jordi Grau-Moya, Marcus Hutter, Tim Genewein

TL;DR: Understanding why prompting is difficult from a Bayesian meta-learning view and using well-controlled experiments on binary sequence

Abstract: Frontier models can be prompted or conditioned to do many tasks, but finding good prompts is not always easy, nor is understanding some performant prompts. We view prompting as finding the best conditioning sequence on a near-optimal sequence predictor. On numerous well-controlled experiments, we show that unintuitive optimal prompts can be better understood given the (not fully accessible) pretraining distribution. Moreover, even using exhaustive search, reliably identifying optimal prompts for practical neural predictors can be surprisingly difficult. Further, we demonstrate that popular prompting methods, such as using demonstrations from the targeted task, can be suboptimal. In addition, we analyze optimal prompts on frontier models, revealing patterns similar to the binary examples and previous findings. Taken together, this work takes an initial step towards understanding optimal prompts, from a statistical and empirical perspective that complements research on frontier models.

Submission Number: 933