Aligning Brains into a Shared Space Improves Their Alignment to Large Language Model

ARNAB BHATTACHARJEE, Zaid Zada, Haocheng Wang, Bobbi Aubrey, Samuel Nastase, Werner K Doyle, Sasha Devore, Patricia Dugan, Daniel Friedman, Orrin Devinsky, Adeen Flinker, Peter Ramadge, Uri Hasson, Ariel Goldstein

Primary Area: applications to neuroscience & cognitive science

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Shared response model, shared space, contextual embeddings, encoding model, Large Language Models

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We show LLM based encoding models can better predict the shared space features estimated from the neural responses of subjects listening to a podcast and projecting the shared space to individual neural space achieves higher encoding performance.

Abstract: The ability of Large Language Models (LLM) to perform remarkably well on various language processing tasks provides a computational modeling framework for studying the neural basis of human language. Recent studies show that the hidden states of the transformer layers of LLM, called contextual embeddings, can predict brain responses through linear encoding models. In this paper, we analyze the neural responses of 8 subjects while they listened to the same 30 minute podcast episode. We use a shared response model to compute the shared information space across subjects and show that LLM-based encoding models achieve significantly better performance in predicting the shared information features than the original brain responses. We also show that we can use this shared space to denoise the individual brain responses by projecting back to the neural space and this process achieves a mean 38% improvement in encoding performance across the subjects. A detailed inspection of this improvement in different brain areas reveals that the improvements are the most prominent in brain areas specialized for language comprehension, specifically in superior temporal gyrus (STG) and inferior frontal gyrus (IFG). Our analysis also shows that the shared space calculated from a group of subjects is generalizable to a new subject. This suggests that the LLM model can be used as a shared linguistic model for how information is shared across brains.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6647