Brain-Informed Language Model Training Enables Scalable and Generalizable Alignment with Human Brain Activity

ICLR 2026 Conference Submission9930 Authors

Keywords: Language models, fMRI, Encoding models, naturalistic stimulus, Representation Learning, Multimodal Learning, Low-Rank Adaptation (LoRA), Transfer Learning, Neuroscience-Informed AI, Scalability, Generalizability

TL;DR: We show that brain-informed training of language models, using dual objectives and scaling across data, models, and subjects, yields robust and generalizable alignment with human brain activity beyond baselines.

Abstract: Language models (LMs) provide rich representational spaces that partially align with neural activity during naturalistic experiences such as movie watching. Yet leveraging brain recordings to actively guide LM training remains underexplored. Here, we address this question by investigating whether functional MRI recordings can guide LLM training by aligning language representations with brain dynamics. Using over 50 hours of fMRI data from six participants watching Friends, plus 10 hours of held-out movies, we augmented pre-trained and randomly initialized LMs with a brain alignment module and compared multiple training strategies. Our results show three main findings. First, brain-informed fine-tuning consistently outperforms text-only baselines and brain-from-scratch models, with voxel-level gains that scale with both model size (GPT-2 124M, LLaMA-2 7B) and training duration (1–40 hours). These improvements generalize across participants and out-of-sample movies, yielding robust cross-subject and cross-stimulus encoding. Second, a dual-objective loss that balances language modeling with brain alignment surpasses brain-only optimization, producing more stable and generalizable encoders. Finally, brain supervision enriches LM representations with multisensory inductive biases: brain-fine-tuned models outperform unimodal baselines on VL-Commonsense, better capturing perceptual and associative properties (e.g., color, shape, co-occurrence) that text-only training underrepresents. Together, these results establish cortical dynamics as an effective supervisory signal, enabling scalable, generalizable, and brain-aligned LMs that internalize aspects of human-like multimodal representation.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 9930