How Neural is a Neural Foundation Model?

ICLR 2026 Conference Submission17665 Authors

Keywords: Foundation Models, Neuroscience, Interpretability, Mouse Visual System, Manifolds

TL;DR: This paper analyzes the internal representations of a neural foundation model using manifolds and a new “tubularity” metric showing divergence from biological representations.

Abstract:

Foundation models have shown remarkable success in fitting biological visual
systems; however, their black-box nature inherently limits their utility for under-
standing brain function. Here, we peek inside a SOTA foundation model of neural
activity (Wang et al., 2025) as a physiologist might, characterizing each ‘neuron’
based on its temporal response properties to parametric stimuli. We analyze how
different stimuli are represented in neural activity space by building decoding man-
ifolds, and we analyze how different neurons are represented in stimulus-response
space by building neural encoding manifolds. We find that the different processing
stages of the model (i.e., the feedforward encoder, recurrent, and readout modules)
each exhibit qualitatively different representational structures in these manifolds.
The recurrent module shows a jump in capabilities over the encoder module by
“pushing apart” the representations of different temporal stimulus patterns. Our
“tubularity” metric quantifies this stimulus-dependent development of neural activ-
ity as biologically plausible. The readout module achieves high fidelity by using
numerous specialized feature maps rather than biologically plausible mechanisms.
Overall, this study provides a window into the inner workings of a prominent neural
foundation model, gaining insights into the biological relevance of its internals
through the novel analysis of its neurons’ joint temporal response patterns. Our
findings suggest design changes that could bring neural foundation models into
closer alignment with biological systems: introducing recurrence in early encoder
stages, and constraining features in the readout module.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 17665