Fusing multimodal neuroimaging data with a variational autoencoder

Eloy Geenjaar, Noah Lewis, Zening Fu, Rohan Venkatdas, Sergey M. Plis, Vince D. Calhoun

Abstract: Neuroimaging studies often collect multimodal data. These modalities contain both shared and mutually exclusive information about the brain. This work aims to find a scalable and interpretable method to fuse the information of multiple neuroimaging modalities into a lower-dimensional latent space using a variational autoencoder (VAE). To assess whether the encoder-decoder pair retains meaningful information, this work evaluates the representations using a schizophrenia classification task. The linear classifier, trained on the representations obtained through dimensionality reduction, achieves an area under the curve of the receiver operating characteristic (ROC-AUC) of 0.8609. Thus, training on a multimodal dataset with functional brain networks and a structural magnetic resonance imaging (sMRI) scan, leads to dimensionality reduction that retains meaningful information. The proposed dimensionality reduction outperforms both early and late fusion principal component analysis on the classification task.Clinical relevance — This work examines the interplay between neuroimaging modalities and their relation to mental disorders. This allows for more complex and rigorous analysis of multimodal neuroimaging data throughout clinical settings.