Decoding Large Language Diffusion Models with Foreseeing Movement

ICLR 2026 Conference Submission18778 Authors

Keywords: Large Language Diffusion Model, Decoding Strategy, Decoding Acceleration

TL;DR: In this paper, we propose FDM (Foreseeing Decoding Method) to more accurately decode outputs of large language diffusion models and FDM-A for the acceleration.

Abstract: Large Language Diffusion Models (LLDMs) benefit from a flexible decoding mechanism that enables parallelized inference and controllable generations over autoregressive models. Yet such flexibility introduces a critical challenge: inference performance becomes highly sensitive to the decoding order of tokens. Existing heuristic methods, however, focus mainly on local effects while overlooking long-term impacts. To address this limitation, we propose the Foreseeing Decoding Method (FDM), a novel approach that integrates both local and global considerations to unlock the full potential, employing a search-based strategy to enable effective optimization in discrete spaces. Furthermore, by analyzing the consistency of chosen tokens in the full decoding process, we develop a variant, FDM with Acceleration (FDM-A), which restricts deep exploration to critical steps identified as the exploration and balance circumantences. Extensive experiments across diverse benchmarks and model architectures validate the scalability of FDM and demonstrate the superior efficiency-performance trade-off achieved by FDM-A. Our work might potentially provide a principled step toward more powerful decoding methods for LLDMs.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 18778