VHEGAN: Variational Hetero-Encoder Randomized GAN for Zero-Shot Learning

Hao Zhang, Bo Chen, Long Tian, Zhengjue Wang, Mingyuan Zhou

Abstract: To extract and relate visual and linguistic concepts from images and textual descriptions for text-based zero-shot learning (ZSL), we develop variational hetero-encoder (VHE) that decodes text via a deep probabilisitic topic model, the variational posterior of whose local latent variables is encoded from an image via a Weibull distribution based inference network. To further improve VHE and add an image generator, we propose VHE randomized generative adversarial net (VHEGAN) that exploits the synergy between VHE and GAN through their shared latent space. After training with a hybrid stochastic-gradient MCMC/variational inference/stochastic gradient descent inference algorithm, VHEGAN can be used in a variety of settings, such as text generation/retrieval conditioning on an image, image generation/retrieval conditioning on a document/image, and generation of text-image pairs. The efficacy of VHEGAN is demonstrated quantitatively with experiments on both conventional and generalized ZSL tasks, and qualitatively on (conditional) image and/or text generation/retrieval.

Keywords: Deep generative models, deep topic modeling, generative adversarial learning, variational encoder, zero-short learning

Data: [CUB-200-2011](https://paperswithcode.com/dataset/cub-200-2011), [Oxford 102 Flower](https://paperswithcode.com/dataset/oxford-102-flower)