Only Brains Align with Brains: Cross-Region Patterns Expose Limits of Normative Models

ICLR 2026 Conference Submission13969 Authors

Keywords: brain alignment, benchmarking, representational similarity analysis, video models

TL;DR: We expose the limits of brain alignment of SOTA video models, and propose a framework based on cross-region alignment patterns in the brain towards more robust and meaningful assessment of brain-model alignment.

Abstract: Normative models of brain regions aim to replicate their representational geometry and are widely used to study neural computation. Model–brain alignment is typically assessed with metrics such as representational similarity analysis (RSA) and linear predictivity (LP). Recent studies, however, show that conclusions from such benchmarks depend strongly on the choice of metric, raising a deeper conceptual problem: what does “brain alignment” truly measure? We address this by testing a broad spectrum of vision models on the BOLD-Moments video fMRI dataset and analyzing the influence of the alignment metric in greater detail. While benchmarks can identify a nominally best model, many other models fall within subject-level variability and are therefore practically equivalent. To move beyond metric dependence, we introduce Alignment Pattern Similarity (APS), a framework that uses brain-to-brain alignment as ground truth for evaluating normative models. For each region, we compare its empirical alignment with other regions against the alignment obtained when replacing that region’s activity with its normative model. Strikingly, while normative models can align well with their target region, their cross-region alignment patterns diverge systematically from those observed in the brain. This reveals a key deficiency: current normative models do not faithfully reproduce brain-to-brain alignment patterns when substituted for real neural data. Furthermore, we show that structural connectivity can predict aspects of these alignment patterns, illustrating how anatomical constraints may additionally guide expectations about functional correspondence. Overall, APS shows great promise to become a principled framework for more robust and biologically meaningful assessments of brain–model alignment.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 13969