On the Entropy Calibration of Language Models

Steven Cao, Gregory Valiant, Percy Liang

Keywords: language models, calibration, entropy, sampling, language model inference

Abstract: Language models are trained with teacher forcing but are used autoregressively, so errors accumulate as more tokens are generated. This issue is well-studied but remains a fundamental problem that harms generation quality. Building on past work, we take the perspective that error accumulation is reflected in the model's entropy, so we can better understand and address it through the lens of entropy calibration. A language model is entropy calibrated if its entropy over generations, i.e. its confidence, matches the log loss it incurs on actual text. First, we find that models are indeed miscalibrated in practice: for base models across a range of sizes, entropy per step increases as more tokens are generated, leading to generations becoming incoherent over time. On the other hand, after instruction tuning, the largest models now have too little entropy (i.e. are overconfident), leading to a lack of diversity in model outputs. From a theoretical perspective, entropy calibration is difficult to attain because it is a global property of the entire generation process, which has an exponentially large output space. Per-step adjustments are tractable but fail to preserve the model's log loss, while global adjustments preserve log loss but are intractable. Our main theoretical contribution is to propose future entropy scaling, an adjustment to the next token probabilities that uses information about the future entropy of each token, i.e. the average entropy of continuations from that token. With additional assumptions, we prove that this adjustment calibrates the model while preserving log loss. While future entropy estimation is expensive, this result suggests that calibration and stabilization of the entropy should be possible without trading off model quality.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8708