What's the Magic Word? A Control Theory of LLM Prompting
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: language models, control theory, LLMs, prompt optimization, alignment, mechanistic interpretability
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: We prove bounds and empirically measure the controllability of LLMs via prompting.
Abstract: Prompt engineering is effective and important in the deployment of LLMs but is poorly understood mathematically.
Here, we formalize prompt engineering as an optimal control problem on LLMs -- where the prompt is considered a control variable for modulating the output distribution of the LLM.
Within this framework, we ask a simple question: given a sequence of tokens, does there always exist a prompt we can prepend that will steer the LLM toward accurately predicting the final token?
We call such an optimal prompt the magic word since prepending the prompt causes the LLM to output the correct answer. If magic words exist, can we find them? If so, what are their properties?
We offer analytic analysis on the controllability of a self-attention head where we prove a bound on controllability as a function of the singular values of its weight matrices.
We take inspiration from control theory to propose a metric called $k - \epsilon$ controllability to characterize LLM steerability.
We compute the $k-\epsilon$ controllability of a panel of large language models, including Falcon-7b, Llama-7b, and Falcon-40b on 5000 WikiText causal language modeling tasks.
Remarkably, we find that magic words of 10 tokens or less exist for over 97\% of WikiText instances surveyed for each model.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 8616
Loading