# Multimodal VAE models

class multimodal_compare.models.mmvae_models.DMVAE(vaes: list, n_latents: int, obj_config: dict, model_config=None)

  Bases: `TorchMMVAE`

  _is_full_backward_hook: bool | None
  forward(x, K=1)

    Forward pass that takes input data and outputs a list of private and shared posteriors, reconstructions and latent samples :param inputs: input data, a list of modalities where missing modalities are replaced with None :type inputs: list :param K: sample K samples from the posterior :type K: int :return: a list of posterior distributions, a list of reconstructions and latent samples :rtype: tuple(list, list, list)

  get_remaining_mods_data(qz_xs: dict, exclude_mod: str)
  logsumexp(x, dim=None, keepdim=False)

    A smooth maximum function :param x: input data :type x: torch.tensor :param dim: dimension :type dim: int :param keepdim: whether to keep shape or squeeze :type keepdim: bool :return: data :rtype: torch.tensor

  objective(mods)

    Objective for the DMVAE model. Source: https://github.com/seqam-lab/

    Parameters:

      data (dict) – input data with modalities as keys

    Return obj:

      dictionary with the obligatory “loss” key on which the model is optimized, plus any other keys that you wish to log

    Rtype obj:

      dict

  property pz_params
  training: bool

class multimodal_compare.models.mmvae_models.MOE(vaes: list, n_latents: int, obj_config: dict, model_config=None)

  Bases: `TorchMMVAE`

  _is_full_backward_hook: bool | None
  forward(x, K=1)

    Forward pass that takes input data and outputs a list of posteriors, reconstructions and latent samples

    Parameters:

      - x (list) – input data, a list of modalities where missing modalities are replaced with None

      - K (int) – sample K samples from the posterior

    Returns:

      a list of posterior distributions, a list of reconstructions and latent samples

    Return type:

      tuple(list, list, list)

  objective(data)

    Objective function for MoE

    Parameters:

      data (dict) – input data with modalities as keys

    Return obj:

      dictionary with the obligatory “loss” key on which the model is optimized, plus any other keys that you wish to log

    Rtype obj:

      dict

  property pz_params
  reconstruct(data, runPath, epoch)

    Reconstruct data for individual experts

    Parameters:

      - data (list) – list of input modalities

      - runPath (str) – path to save data to

      - epoch (str) – current epoch to name the data

  training: bool

class multimodal_compare.models.mmvae_models.MoPOE(vaes: list, n_latents: int, obj_config: dict, model_config=None)

  Bases: `TorchMMVAE`

  _is_full_backward_hook: bool | None
  forward(inputs, K=1)

    Forward pass that takes input data and outputs a list of posteriors, reconstructions and latent samples :param inputs: input data, a list of modalities where missing modalities are replaced with None :type inputs: list :param K: sample K samples from the posterior :type K: int :return: a list of posterior distributions, a list of reconstructions and latent samples :rtype: tuple(list, list, list)

  mixture_component_selection(mus, logvars, w_modalities=None)
  modality_mixing(input_batch)

    Mix the encoded distributions according to the chosen approach

    Parameters:

      mods (dict) – qz_xs dictionary with modalities as keys and distribution parameters as values

    Returns:

      latent samples dictionary with modalities as keys and latent sample tensors as values

    Return type:

      dict

  moe_fusion(mus, logvars, weights)
  objective(mods)

    Objective function for MoPoE. Computes GENERALIZED MULTIMODAL ELBO https://arxiv.org/pdf/2105.02470.pdf

    Parameters:

      data (dict) – input data with modalities as keys

    Return obj:

      dictionary with the obligatory “loss” key on which the model is optimized, plus any other keys that you wish to log

    Rtype obj:

      dict

  poe_fusion(mus, logvars)
  property pz_params
  reparameterize(mu, logvar)
  reweight_weights(w)
  set_subsets()

    powerset([1,2,3]) –> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)

  training: bool

class multimodal_compare.models.mmvae_models.POE(vaes: list, n_latents: int, obj_config: dict, model_config=None)

  Bases: `TorchMMVAE`

  _is_full_backward_hook: bool | None
  forward(inputs, K=1)

    Forward pass that takes input data and outputs a dict with posteriors, reconstructions and latent samples :param inputs: input data, a dict of modalities where missing modalities are replaced with None :type inputs: dict :param K: sample K samples from the posterior :type K: int :return: dict where keys are modalities and values are a named tuple :rtype: dict

  modality_mixing(x)

    Inference module, calculates the joint posterior :param inputs: input data, a dict of modalities where missing modalities are replaced with None :type inputs: dict :param K: sample K samples from the posterior :type K: int :return: joint posterior and individual posteriors :rtype: tuple(torch.tensor, torch.tensor, list, list)

  objective(mods)

    Objective function for PoE

    Parameters:

      data (dict) – input data with modalities as keys

    Return obj:

      dictionary with the obligatory “loss” key on which the model is optimized, plus any other keys that you wish to log

    Rtype obj:

      dict

  prior_expert(size, use_cuda=False)

    Universal prior expert. Here we use a spherical Gaussian: N(0, 1).

    Parameters:

      - size (int) – dimensionality of the Gaussian

      - use_cuda (boolean) – cast CUDA on variables

    Returns:

      mean and logvar of the expert

    Return type:

      tuple

  property pz_params
  training: bool