Forcing Diffuse Distributions out of Language Models

Yiming Zhang, Avi Schwarzschild, Nicholas Carlini, J Zico Kolter, Daphne Ippolito

Research Area: Inference algorithms for LMs

Keywords: diversity, synthetic dataset generation, parameter-efficient fine-tuning

TL;DR: This paper proposes methods to enhance language models for generating diverse outputs, crucial for tasks like dataset construction.

Abstract:

Despite being trained specifically to follow user instructions, today’s instruction-tuned
language models perform poorly when instructed to produce random outputs.
For example, when prompted to pick a number uniformly between one and
ten Llama-2-13B-chat disproportionately favors the number five, and when tasked
with picking a first name at random, Mistral-7B-Instruct chooses Avery 40 times
more often than we would expect based on the U.S. population. When these language
models are used for real-world tasks where diversity of outputs is crucial,
such as language model assisted dataset construction, their inability to produce
diffuse distributions over valid choices is a major hurdle. In this work, we propose
a fine-tuning method that encourages language models to output distributions that
are diffuse over valid outcomes. The methods we introduce generalize across a
variety of tasks and distributions and make large language models practical for
synthetic dataset generation with little human intervention.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html

Author Guide: I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html

Submission Number: 977