Diverse Text Decoding via Iterative Reweighting

ICLR 2026 Conference Submission15222 Authors

Keywords: Natural Language Processing, Large Language Models, Text Generation, Decoding Method

TL;DR: We propose OverRIDE, a dynamic decoding method that improves text generation diversity.

Abstract: Recent advances in large language models (LLMs) have led to impressive results in text generation. However, current decoding methods still lack diversity when combined with popular sampling techniques. We propose a Reweighting-based Iterative DEcoding (OverRIDE) approach that dynamically adjusts the decoding process with history responses. Our method fine-tunes auxiliary output heads iteratively on previously generated sequences to capture and suppress semantic patterns that appear in the history responses. This inference-time training process only incurs minimal loss of efficiency. We conduct extensive experiments on various tasks, including code generation, mathematical reasoning and story generation, demonstrating that OverRIDE increases output diversity while maintaining quality. We implement OverRIDE on LLM serving systems like vLLM, achieving a 6.4\% throughput loss for 72B models under parallel decoding.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 15222