Can One Token Make All the Difference? Forking Paths in Autoregressive Text Generation

Eric J Bigelow, Ari Holtzman, Hidenori Tanaka, Tomer Ullman

Track: Extended abstract

Keywords: Large Language Models, Interpretability, Uncertainty

TL;DR: We present a novel black-box approach to analyzing uncertainty in text generation and demonstrate how language models can suddenly transition from one pattern to another over the course of text generation.

Abstract: Analyzing uncertainty is essential for properly evaluating and safely using Large Language Models (LLMs). When an LLM generates some text, we might ask what alternate text could it have produced during the decoding process? Would an LLM generate a very different response if even just one token was sampled differently? We present a novel approach to understanding uncertainty in autoregressive text generation through a black-box methodology we call Forking Paths Analysis. Our method involves a multi-stage LLM sampling architecture, followed by the aggregation of samples into a multivariate time series representing the outcome distribution, and concludes with the application of change point detection to pinpoint tokens where the distribution shifts suddenly. We apply Forking Paths Analysis to questions from 7 datasets commonly used to evaluate LLMs. Our analysis reveals striking uncertainty dynamics over the course of text generation, including dramatic change points where one token can make all the difference.

Submission Number: 72