Two-Stage Diffusion Models: Better Image Synthesis by Explicitly Modeling Semantics
Keywords: generative modeling, diffusion models
TL;DR: We advocate for two-stage diffusion models that first sample an embedding describing the semantics of an image, and then the pixel values.
Abstract: Recent progress with conditional image diffusion models has been stunning, and this holds true whether we are speaking about models conditioned on a text description, a scene layout, or a sketch. Unconditional image diffusion models are also improving but lag behind, as do diffusion models which are conditioned on lower-dimensional features like class labels. We advocate for a simple method that leverages this phenomenon for better unconditional generative modeling. In particular, we suggest a two-stage sampling procedure. In the first stage we sample an embedding describing the semantic content of the image. In the second stage we use a conditional image diffusion model to sample the image conditioned on this embedding, and then discard the embedding. The combined model can therefore leverage the power of conditional diffusion models on the unconditional generation task, achieving large improvements in unconditional image generation. The same method can be generalized to yield similar improvements for image generation conditioned on a low-dimensional signal like a class label.
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 7930
Loading