Oracle Guided Image Synthesis with Relative QueriesDownload PDF

Published: 29 Mar 2022, Last Modified: 22 Oct 2023ICLR 2022 DGM4HSD workshop PosterReaders: Everyone
Keywords: Variational Autoencoder, Image Synthesis, Relative Attributes, Paired Comparisons, Triplet Loss, Structured latent representations, Human-Computer Interaction
TL;DR: We describe a method for performing guided image synthesis with a VAE by asking users queries of the form "do you prefer image A or image B?"
Abstract: Isolating and controlling specific features in the outputs of generative models in a user-friendly way is a difficult and open-ended problem. We develop techniques that allow a user to generate an image they are envisioning in their head by answering a sequence of relative queries of the form \textit{``do you prefer image $a$ or image $b$?''} Our framework consists of a Conditional VAE that uses the collected relative queries to partition the latent space into preference-relevant features and non-preference-relevant features. We then use the user's responses to relative queries to determine the preference-relevant features that correspond to their envisioned output image. Additionally, we develop techniques for modeling the uncertainty in images' predicted preference-relevant features, allowing our framework to generalize to scenarios in which the relative query training set contains noise.
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 1 code implementation](
2 Replies