Rethinking Diffusion Model in High Dimension

Zhenxin Zheng, Zhenjie Zheng

Keywords: Diffusion Model, High Dimension, Diffusion Model Inference Method, Curse of Dimensionality

TL;DR: This paper attempts to analyze the working principles of high-dimensional diffusion models.

Abstract: Curse of Dimensionality is an unavoidable challenge in statistical probability models, yet diffusion models seem to overcome this limitation, achieving impressive results in high-dimensional data generation. Diffusion models assume that they can learn the statistical properties of the underlying probability distribution, enabling sampling from this distribution to generate realistic samples. But is this really how they work? To address this question, this paper conducts a detailed analysis of the objective function and inference methods of diffusion models, leading to several important conclusions that help answer the above question: 1) In high-dimensional sparse scenarios, the target of the objective function fitting degrades from a weighted sum of multiple samples to a single sample. 2) The mainstream inference methods can all be represented within a simple unified framework, without requiring statistical concepts such as Markov chain and SDE, while aligning with the degraded objective function. 3) Guided by this simple framework, more efficient inference methods can be discovered. Code is available at Supplementary Material.

Primary Area: Probabilistic methods (e.g., variational inference, causal inference, Gaussian processes)

Submission Number: 16851