Capturing Label Characteristics in VAEs

Tom Joy, Sebastian Schmon, Philip Torr, Siddharth N, Tom Rainforth

Keywords: variational autoencoder, representation learning, deep generative models

Abstract: We present a principled approach to incorporating labels in variational autoencoders (VAEs) that captures the rich characteristic information associated with those labels. While prior work has typically conflated these by learning latent variables that directly correspond to label values, we argue this is contrary to the intended effect of supervision in VAEs—capturing rich label characteristics with the latents. For example, we may want to capture the characteristics of a face that make it look young, rather than just the age of the person. To this end, we develop a novel VAE model, the characteristic capturing VAE (CCVAE), which “reparameterizes” supervision through auxiliary variables and a concomitant variational objective. Through judicious structuring of mappings between latent and auxiliary variables, we show that the CCVAE can effectively learn meaningful representations of the characteristics of interest across a variety of supervision schemes. In particular, we show that the CCVAE allows for more effective and more general interventions to be performed, such as smooth traversals within the characteristics for a given label, diverse conditional generation, and transferring characteristics across datapoints.

One-sentence Summary: We present a principled approach to incorporating labels in VAEs that captures the rich characteristic information associated with those labels.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Code: [ thwjoy/ccvae](https://github.com/thwjoy/ccvae) + [ 1 community implementation](https://paperswithcode.com/paper/?openreview=wQRlSUZ5V7B)

Data: [CelebA](https://paperswithcode.com/dataset/celeba), [CheXpert](https://paperswithcode.com/dataset/chexpert)