Towards foundation models of naturalistic collective social‑neural dynamics

Asaf Benjamin, Sergey Anpilov, Omer Izhaki, Maya Sheffer, Tommaso Biagini, Yair Shemesh, Alon Rubin, Inbar Saraf-Sinik, Mackenzie W Mathis, Alexander Mathis, Alon Chen, Ofer Yizhar

Keywords: Contrastive learning, Self-supervised learning, Neuroscience, Foundation models, explainable AI

TL;DR: Self-supervised foundation models of naturalistic social interactions improve neural predictions

Abstract: Foundation models for the brain and body (FMBB) can transform neuroscience by improving generalization, standardization and reproducibility. However, building generally useful FMBB requires large-scale, high-quality datasets, which capture the immense complexity and variability of naturalistic interactions. Unfortunately, measuring neural activity during such interactions is extremely challenging, especially with high spatiotemporal resolution and in human subjects. Therefore, we have collected a large and unique dataset comprising 17 groups of three-four mice, freely interacting in an enriched environment under continuous video monitoring for one week. We used wireless neural loggers to electrophysiologically record medial prefrontal cortex (mPFC) activity with single-spike, single-unit resolution from all the group members simultaneously, and we systematically perturbed this neural activity using wireless optogenetics to measure its behavioral effects. To study different levels of behavior and their neural representations, we established an extensive, carefully curated and highly accurate preprocessing pipeline, including spike sorting, 3D pose estimation, interpretable behavioral feature extraction, high‑level behavior classification, and social dominance hierarchy (SDH) extraction. We then trained foundation models using self‑supervised representation learning with CEBRA on the behavioral data, pooled across sessions and animals. We devised a custom method of feature partitioning to make the contrastive learning task more challenging and show that using the learned embeddings instead of the original features as inputs to downstream models trained to predict neural activity significantly improves their performance. We then use feature attribution methods to show how this can complement classical analysis of neural tuning. Taken together, this work paves a path towards building generally useful FMBB during naturalistic social interactions.

Submission Number: 19