Energy-Based Multimodal VAEs

Anonymous

Keywords: multimodal VAE, energy-based models, multimodal generative model

Abstract:

Multimodal VAEs are a promising class of multimodal generative models that constructs a tractable posterior over the latent space given all modalities. Daunhawer et al. (2022) show that the generative quality of each modality drops as we increase the number of modalities.
In this work, we take another direction to address the generative quality of multimodal VAEs by jointly modeling the latent space of unimodal VAEs using energy-based models (EBMs). The role of EBM is to enforce multimodal coherence by learning the correlation among the latent variables. Therefore, our model enjoys the high generative quality of unimodal VAEs while maintaining coherence across different modalities.

Student Paper: Yes