Zero-shot neural predictivity in human prefrontal cortex with a massively multi-task multimodal transformer

Lucas Gomez, Aziz Ktari, Mark Bai, Pouya Bashivan

TL;DR: Train a large multi-task neural network model on visual working memory tasks. Analyse the neural similarity of thes model with fMRI activity of human PFC. Investigate representational hierarchy of the PFC using model layer-to-roi mappings.

Abstract: Working memory supports a broad spectrum of behaviors and higher cognitive abilities, with the prefrontal cortex playing a central role in this capacity. Although prior work has identified which brain regions are engaged in specific working memory tasks, and in some cases how they contribute, we still lack a general framework that can predict which regions will be recruited in novel tasks, what information they represent, and the computations they perform. To address this gap, we trained a single neural network on millions of visual decision-making tasks with sensory-realistic inputs, aiming to build a generalized model of working memory. We evaluated the model against an fMRI dataset spanning 12 tasks and hundreds of distinct conditions, testing its ability to capture neural activity across the brain, with a focus on the prefrontal cortex. Our results show that large models trained on a broad distribution of tasks can predict brain activity zero-shot, outperforming even models trained directly on the target tasks. This ability improves further with model size, which consistently enhances prediction accuracy. Furthermore, analyses of layer-to-region correspondences largely conformed with the theories of hierarchical organization along the rostro-caudal axis of the prefrontal cortex. These findings suggest that neural network models hold significant potential not only for simulating neural activity in regions previously difficult to model, but also for revealing how the brain encodes, organizes, and manipulates task information during working memory.

Length: long paper (up to 8 pages)

Domain: methods

Author List Check: The author list is correctly ordered and I understand that additions and removals will not be allowed after the abstract submission deadline.

Anonymization Check: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and URLs that point to identifying information.

Submission Number: 30