# MultimodalVAE class

class multimodal_compare.models.trainer.MultimodalVAE(cfg, feature_dims: dict)

  Bases: `LightningModule`

  Multimodal VAE trainer common for all architectures. Configures, trains and tests the model.

  Parameters:

    - feature_dims (dict) – dictionary with feature dimensions of training data

    - cfg (object) – instance of Config class

  analyse_data(data=None, labels=None, num_samples=250, path_name='', savedir=None, split='val')

    Encodes data and plots T-SNE. If no data is passed, a dataloader (based on split=”val”/”test”) will be used.

    Parameters:

      - data (torch.tensor) – test data

      - labels (list) – labels for the data for labelled T-SNE (optional) - list of strings

      - num_samples (int) – number of samples to use for visualization

      - path_name (str) – label under which to save the visualizations

      - savedir (str) – where to save the reconstructions

      - split (str) – val/test, whether to take samples from test or validation dataloader

  check_config(cfg)

    Creates a Config class out of the provided argument parser

    Parameters:

      cfg ((argparse.ArgumentParser, str)) – argument parser or str path to config

    Returns:

      Config instance

    Return type:

      object

  configure_optimizers()

    Sets up the optimizer specified in the config

  property datamod

    When the class is used for inference, there is no pl trainer module

    Returns:

      an instance of DataModule class

    Return type:

  eval_forward(data)

    Forward pass used outside training, e.g. during evaluation

  get_mod_names()

    Creates a dictionary with modality numbers and their names based on dataset

    Returns:

      Dict with modality numbers as keys and names as values

    Return type:

      dict

  get_model()

    Sets up the model according to the config file

  save_joint_samples(num_samples=16, savedir=None, traversals=False)

    Generate joint samples from random vectors and save them

    Parameters:

      - num_samples (int) – number of samples to generate

      - savedir (str) – where to save the reconstructions

      - traversals (bool) – whether to make traversals for each dimension (True) or randomly sample latents (False)

  save_reconstructions(num_samples=10, savedir=None, split='val')

    Reconstructs data and saves output, also iterates over missing modalities on the input to cross-generate

    Parameters:

      - num_samples (int) – number of samples to take from the dataloader for reconstruction

      - savedir (str) – where to save the reconstructions

      - split (str) – val/test, whether to take samples from test or validation dataloader

  test_epoch_end(outputs)

    Visualizations to make at the end of the testing epoch

  test_step(test_batch, batch_idx)

    Iterates over the test loader (if test data is provided, otherwise val loader)

  training_step(train_batch, batch_idx)

    Iterates over the train loader

  validation_epoch_end(outputs)

    Save visualizations at the end of validation epoch

    Parameters:

      outputs (torch.tensor) – Loss that comes from validation_step

  validation_step(val_batch, batch_idx)

    Iterates over the val loader