The emergence of the left-right asymmetry in predicting brain activity from LLMs' representations specifically correlates with their formal linguistic competence

Laurent Bonnasse-Gahot, Christophe Pallier

Keywords: large language models, training, language processing, brain lateralization, fMRI, phase transition, syntax

TL;DR: As training progresses, the internal representations of an LLM become increasingly better at predicting brain activity, with a sudden increase in asymmetry between left and right hemispheres coinciding with the LLM's acquisition of formal competence.

Abstract: When humans and large language models (LLMs) process the same text, activations in the LLMs correlate with brain activity measured, e.g., with functional magnetic resonance imaging (fMRI). Moreover, as the training of an LLM progresses, the performance in predicting brain activity from its internal activations improves more in the left hemisphere than in the right one. The aim of the present work is to understand which kind of competence acquired by the LLMs underlies the emergence of this left-right asymmetry. Using the OLMo-2 7B language model at various training checkpoints and fMRI data from English participants, we compare the evolution of the left-right asymmetry in brain scores alongside performance on several benchmarks. We observe that the asymmetry co-emerges with the formal linguistic abilities of the LLM. These abilities are demonstrated in two ways: by the model's capacity to assign a higher probability to an acceptable sentence than to a grammatically unacceptable one within a minimal contrasting pair, or its ability to produce well-formed text. On the opposite, the left-right asymmetry does not correlate with the performance on arithmetic or Dyck language tasks; nor with text-based tasks involving world knowledge and reasoning. We generalize these results to another family of LLMs (Pythia) and another language, namely French. Our observations indicate that the left-right asymmetry in brain predictivity matches the progress in formal linguistic competence (knowledge of linguistic patterns).

Primary Area: applications to neuroscience & cognitive science

Submission Number: 18896