Unity by Diversity: Improved Representation Learning in Multimodal VAEs

Thomas M. Sutter, Yang Meng, Norbert Fortin, Julia E Vogt, Babak Shahbaba, Stephan Mandt

Keywords: multimodal VAE, representation learning, data-dependent prior, vamp-prior

TL;DR: We propose a novel multimodal VAE that learns from data using a data-dependent mixture-of-experts prior for soft-sharing of information between modalities

Abstract:

Variational Autoencoders for multimodal data hold promise for many tasks in data analysis, such as representation learning, conditional generation, and imputation.
Current architectures either share the encoder output, decoder input, or both across modalities to learn a shared representation.
Such architectures impose hard constraints on the model.
In this work, we show that a better latent representation can be obtained by replacing these hard constraints with a soft constraint. We propose a new mixture-of-experts prior, softly guiding each modality's latent representation towards a shared aggregate posterior.
This approach results in a superior latent representation and allows each encoding to preserve information from its uncompressed original features better. In extensive experiments on multiple benchmark datasets and a challenging real-world neuroscience data set, we show improved learned latent representations and imputation of missing data modalities compared to existing methods.

Submission Number: 10