Generation, Reconstruction, Representation All-in-One: A Joint Autoencoding Diffusion Model

Guangyi Liu, Yu Wang, Zeyu Feng, Qiyu Wu, Liping Tang, Yuan Gao, Zhen Li, Shuguang Cui, Eric Xing, Zichao Yang, Zhiting Hu

Keywords: diffusion model, generative model, latent model

Abstract: The vast applications of deep generative models are founded on the premise of three fundamental capabilities: generating new instances (e.g., image/text synthesis and molecule design), reconstructing inputs (e.g., data editing and restoration), and learning latent representations (e.g., structure discovery and downstream classification). Existing model families, including Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), autoregressive models, and diffusion models, generally excel in specific capabilities but fall short in others. We introduce Joint Autoencoding Diffusion (JEDI), a new generative framework that unifies all three core capabilities, offering versatile applications and strong performance in a single model. Specifically, JEDI generalizes the noising/denoising transformations (based on simple Gaussian noise) in diffusion process by introducing parameterized encoder/decoder transformations between raw data and compact representations. Crucially, the encoder/decoder parameters are learned jointly with all other diffusion model parameters under the standard probabilistic diffusion formalism. This results in a model that not only inherits the strong generation abilities of diffusion models but also enables compact data representation and faithful reconstruction. Additionally, by choosing appropriate encoder/decoder, JEDI can naturally accommodate discrete data (such as text and protein sequences) which have been difficult for diffusion models. Extensive experiments across different data modalities, including images, text, and proteins, demonstrate JEDI's general applicability to diverse tasks and strong improvement over existing specialized deep generative models.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8162