The One Where They Brain-Tune for Social Cognition: Multi-Modal Brain-Tuning on Friends

Nico Policzer, Cameron Braunstein, Mariya Toneva

Keywords: braintuning, multimodal models, video models, brain alignment, fMRI, social cognition, sarcasm detection, emotion detection

TL;DR: We brain-tune the TVLT Model and assess its sarcasm and emotion detection capabilities.

Abstract: Recent studies on audio models show brain-tuning–fine-tuning models to better predict corresponding fMRI activity–improves brain alignment and increases performance on downstream semantic and audio tasks. We extend this approach to a multimodal audio-video model to enhance social cognition, targeting the Superior Temporal Sulcus (STS), a key region for social processing, while subjects watch Friends. We find significant increases in brain alignment to the STS and an adjacent ROI, as well as improvements to a social cognition task related to the training data— sarcasm detection in sitcoms. In summary, our study extends brain-tuning to the multi-modal domain, demonstrating improvements to a downstream task after tuning to a relevant functional region.

Submission Number: 8