Neural Decoding through Multi-subject Class-conditional Hyperalignment

Brian Schetzsle, Yang Meng, Norbert J. Fortin, Babak Shahbaba

Keywords: latent representation learning, supervised contrastive learning, hyperalignment

TL;DR: We greatly expand the scope of multi-subject neural datasets available for latent representational alignment by introducing a class-conditional hyperalignment method which can be applied directly to experimental data.

Abstract: Understanding brain dynamics in multi-subject studies is challenging, as each individual exhibits unique neural patterns. Such variability complicates the identification of shared task-related dynamics without carefully accounting for meaningful individual differences. Typical analyses involve fitting subject-specific models separately and aggregating results post hoc. This approach, however, precludes the possibility of information sharing across the models. Hyperalignment methods resolve this by mapping subject-specific responses into a shared latent representational space, but typically require a secondary dataset to learn these mappings by exposing all subjects to an identical, rich and evocative stimulus, such as watching an exciting movie. These datasets are costly to collect and understandably infeasible in nonhuman studies. An alignment method for multi-subject studies that can be applied directly to the primary dataset would be of immense value. To this end, we introduce the Multi-Subject Class-Conditional Hyperalignment ($\mathbf{MuSCH}$) model which learns aligned latent embeddings of multi-subject data by leveraging class labels available from the experimental protocol of the primary dataset itself. $\mathbf{MuSCH}$ trains subject-specific encoder networks using a novel Supervised Contrastive Learning framework which simultaneously makes same-class embeddings similar and different-class embeddings dissimilar across subjects. Using both simulation studies and a real memory experiment, we demonstrate how principled information sharing improves the performance of a downstream neural decoding task. Furthermore, by modulating signal strength in the simulated dataset, we show that classification improvements are especially pronounced in regimes with weak signals, a situation commonly encountered in neuroscience investigations. $\mathbf{MuSCH}$ obviates traditional hyperalignment's onerous prerequisite of a secondary alignment dataset, extending the promise of a single robust and generalizable model to any labeled, multi-subject dataset where subject-specific distortions prevent a joint analysis.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 9772