Quantifying Information Flow in Diffusion Models: Entropy-Guided Noise Scheduling and Mutual Information Evaluation

Yun Chen, Guilong Xiao, Shaopeng Nie, Zhuo-Xu Cui

Keywords: Diffusion Models, Noise Scheme, Entropy, Mutual Information

TL;DR: This paper introduces an entropy-guided noise scheduling strategy and a mutual-information-based evaluation framework for diffusion models.

Abstract: Denoising diffusion probabilistic models (DDPMs) and their variants have achieved strong performance across a wide range of tasks, from image restoration to text-to-image generation. Despite these successes, the interplay between timesteps and noise schedules in the diffusion process remains poorly understood. In particular, it is unclear how these factors shape information flow and influence the quality of the final output. This paper investigates diffusion models through the lens of information theory. We introduce an entropy-guided noise scheduling strategy and a mutual-information-based evaluation framework. First, leveraging the differential entropy of Gaussian distributions, we develop a method to compute entropy values of noisy images that are consistent with the diffusion process. Building on this, we design an entropy-guided scheduling strategy to explicitly link timesteps with noise levels during the forward process. Finally, we propose a mutual-information-based evaluation metric to assess the image restoration ability of DDPMs. Experiments on MNIST and Fashion-MNIST demonstrate the feasibility of quantifying and guiding information flow in diffusion models.

Primary Area: interpretability and explainable AI

Submission Number: 24344