Prompting as Scientific Inquiry

Ari Holtzman, Chenhao Tan

Keywords: prompting, capabilities discovery, mechanistic interpretability

TL;DR: Prompting is legitimate behavioral science that has unlocked most major LLM capabilities, not the unscientific "alchemy" it's often dismissed as.

Abstract: Prompting is the primary method by which we study and control large language models. It is also one of the most powerful: nearly every major capability attributed to LLMs—few-shot learning, chain-of-thought, constitutional AI—was first unlocked through prompting. Yet prompting is rarely treated as science and is frequently frowned upon as alchemy. We argue that this is a category error. If we treat LLMs as a new kind of organism—complex, opaque, and trained rather than programmed—then prompting is not a workaround. It is behavioral science. Mechanistic interpretability peers into the neural substrate, prompting probes the model in its native interface: language. We argue that prompting is not inferior, but rather a key component in the science of LLMs.

Lay Summary:

Prompting is not a workaround—it is the primary way we study and steer large language models (LLMs). Many headline abilities—few-shot learning, chain-of-thought, and constitutional AI—were first unlocked by carefully phrasing inputs. Treating LLMs as organisms to be probed, prompting becomes behavioral science that complements mechanistic interpretability.

We define prompting as interacting with LLMs in natural language and observing their outputs or probabilities. It is powerful for two reasons: (1) people grasp the structure of language; and (2) today’s models were discovered, not designed, so language is our most direct handle for revealing novel behavior.

Prompting and interpretability intervene at different levels: interpretability alters internals; prompting perturbs inputs at the model’s native interface to establish causal links between context and behavior. Practically, prompting scales, maps broad behaviors, and seeds hypotheses that mechanistic work can later confirm—forming a bidirectional loop from behavior to mechanism and back.

The upshot: to understand and control LLMs, we should treat prompt design and analysis as first-class scientific methods, used alongside circuit-level tools.

Submission Number: 564