Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models

Yuge Shi, N Siddharth, Brooks Paige, Philip Torr

Abstract: Learning generative models that span multiple data modalities, such as vision and language, is often motivated by the desire to learn more useful, generalisable representations that faithfully capture common underlying factors between the modalities. In this work, we characterise successful learning of such models as the fulfilment of four criteria; i) implicit latent decomposition into shared and private subspaces, ii) coherent joint generation over all modalities, iii) coherent cross-generation across individual modalities, and iv) improved model learning for individual modalities through multi-modal integration. We see that prior approaches largely ignore characterisation of the learnt representations and evaluation of how the joint generations are related, focusing primarily on cross-modal generation, with a handful of approaches also exploring the value of multi-modal integration for a single modality. Here, we propose a mixture-of-experts variational autoencoder (VAE) for learning multi-modal generative models on different sets of modalities, including a challenging image↔language dataset, and demonstrate its ability to satisfy all four criteria, both qualitatively and quantitatively.

Code Link: https://github.com/iffsid/mmvae

CMT Num: 9192