What Brain Data Adds to Language Model Training

Gabriele Merlin, Omer Moussa, Mariya Toneva

Keywords: language models, brain alignment, brain tuning, linguistic competence, neuroscience, fMRI

Abstract: Brain-tuning language models (LMs)---fine-tuning LMs to predict brain recordings elicited by linguistic stimuli---has been proposed as a promising way to align LMs closer to the human brain, with recent work reporting gains on a small number of downstream tasks. However, it remains unclear what benefits brain data provide beyond those obtainable from further training on the same underlying linguistic input, and whether such benefits generalize across tasks. Here, we present a comprehensive evaluation of jointly-tuned LMs, trained on both brain recordings and text-based stimuli, brain-tuned LMs and LMs tuned only on text-based stimuli (i.e., stimulus-tuned LMs). We compare models across a diverse suite of downstream linguistic tasks. We find that jointly-tuned LMs outperform other fine-tuned and pretrained models, and that brain-tuned LMs outperform stimulus-tuned LMs, demonstrating the richness of brain data as an additional training signal for LMs.

Scope Confirmation: To the best of my judgment, this submission falls within the scope of CoNLL.

Primary Area Selection: Language and the Brain

Secondary Area Selection: Computational Psycholinguistics, Cognition and Linguistics

Use Of Generative Artificial Intelligence Tools: Yes, for editing/proofreading the manuscript

Data Collection From Human Subjects: No

Submission Type: Archival: I certify that the submission has not been previously published, nor is the material in it under review by another journal or conference. Further, no material in it will be submitted for review at another conference or journal while under review by CoNLL 2026.

Submission Number: 226