Multimodal neural networks better explain multivoxel patterns in the hippocampusDownload PDF

Oct 12, 2021 (edited Dec 10, 2021)SVRHM 2021 OralReaders: Everyone
  • Keywords: multimodal networks, concept cells, hippocampus, fMRI, deep learning, neuroscience
  • TL;DR: Using publicly available fMRI data, we investigate different publicly available models and demonstrate that multimodal networks are better in explaining multivoxel activity in the human hippocampus.
  • Abstract: The human hippocampus possesses "concept cells", neurons that fire when presented with stimuli belonging to a specific concept, regardless of the modality. Recently, similar concept cells were discovered in a multimodal network called CLIP [1].Here, we ask whether CLIP can explain the fMRI activity of the human hippocampus better than a purely visual (or linguistic) model. We extend our analysis to a range of publicly available uni- and multi-modal models. We demonstrate that ``multimodality'' stands out as a key component when assessing the ability of a network to explain the multivoxel activity in the hippocampus.
  • Supplementary Material: zip
5 Replies