Conditional Image Generation by Conditioning Variational Auto-EncodersDownload PDF

Published: 28 Jan 2022, Last Modified: 13 Feb 2023ICLR 2022 PosterReaders: Everyone
Keywords: variational auto-encoders, Bayesian inference, variational inference, amortized inference, image completion
Abstract: We present a conditional variational auto-encoder (VAE) which, to avoid the substantial cost of training from scratch, uses an architecture and training objective capable of leveraging a foundation model in the form of a pretrained unconditional VAE. To train the conditional VAE, we only need to train an artifact to perform amortized inference over the unconditional VAE's latent variables given a conditioning input. We demonstrate our approach on tasks including image inpainting, for which it outperforms state-of-the-art GAN-based approaches at faithfully representing the inherent uncertainty. We conclude by describing a possible application of our inpainting model, in which it is used to perform Bayesian experimental design for the purpose of guiding a sensor.
One-sentence Summary: We create fast-to-train conditional VAEs using amortized inference in pretrained unconditional VAEs, and demonstrate diverse samples on image completion tasks.
Supplementary Material: zip
16 Replies