Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

ICLR 2026 Conference Submission3534 Authors

Keywords: LLM Diversity, Mode Collapse, Creative Writing, Dialogue Simulation, Human Preference Bias

TL;DR: We identify typicality bias in human preferences as a key cause of LLM diversity drop in RLHF; grounded in the theoretical insights, we propose Verbalized Sampling, a simple yet effective prompting method that improves LLM diversity across tasks.

Abstract: Post-training alignment often reduces LLM diversity, leading to a phenomenon known as mode collapse. Unlike prior work that attributes this effect to algorithmic limitations, we identify a fundamental, pervasive data-level driver: typicality bias in preference data, whereby annotators systematically favor familiar text as a result of well-established findings in cognitive psychology. We formalize this bias theoretically, verify it on preference datasets empirically, and show that it plays a central role in mode collapse. Motivated by this analysis, we introduce Verbalized Sampling (VS), a simple, training-free prompting strategy to circumvent mode collapse. VS prompts the model to verbalize a probability distribution over a set of responses (e.g., "Generate 5 jokes about coffee and their corresponding probabilities"), which relieves the pressure to produce a single "typical" answer. Experiments show that VS significantly improves performance across creative writing (poems, stories, jokes), social dialogue simulation, synthetic data generation, and open-ended QA, without sacrificing safety and factual accuracy. For instance, in creative writing, VS increases diversity by 1.6-2.1x compared to direct prompting. We further observe an emergent trend that more capable models benefit more from VS. In sum, our work provides a new data-centric perspective on mode collapse and a practical inference-time remedy that helps unlock pre-trained generative diversity.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 3534