Score-based Idempotent Distillation of Diffusion Models

ICLR 2026 Conference Submission21133 Authors

Keywords: diffusion model, idempotent models, generative models

Abstract: Idempotent generative networks (IGNs) are a new line of generative models based on the idea of idempotent mapping to a target manifold. IGNs support both single-and multi-step generation, allowing for a flexible trade-off between computational cost and sample quality. But similar to Generative Adversarial Networks (GANs), conventional IGNs require adversarial training and are prone to training instabilities and mode collapse. Diffusion and score-based models are popular approaches to generative modeling that iteratively transport samples from one distribution, usually a Gaussian, to a target data distribution. These models have gained popularity due to their stable training dynamics and high-fidelity generation quality. However, this stability and quality come at the cost of high computational cost, as the data must be transported incrementally along the entire trajectory. New sampling methods, model distillation, and consistency models have been developed to reduce the sampling cost and even perform one-shot sampling from diffusion models. In this work, we unite diffusion and idempotent models by training idempotent models through distillation from diffusion models' scores. Our proposed method to train IGNs is highly stable and does not require adversarial losses. We provide a theoretical analysis of our proposed score-based training methods. We empirically show that idempotent networks can be effectively distilled from a pre-trained diffusion model, enabling faster inference compared to iterative score-based models. Like IGNs and score-based models, SIGNs can perform multi-step sampling, allowing users to trade off quality for efficiency. As these models operate directly on the source domain, they can project corrupted or alternate distributions back onto the target manifold, enabling zero-shot editing of inputs. We validate our models on a simple multi-modal dataset as well as multiple image datasets, achieving state-of-the-art results for idempotent models on the CIFAR and CelebA datasets.

Primary Area: generative models

Submission Number: 21133