SYNAPSE: A Multi-Modal Framework for Interpretable Neural Decoding Using Vision-Language Foundation Models

ICML 2025 Workshop FM4LS Submission59 Authors

Keywords: Neural decoding, semantic feature extraction, cross-model embeddings, GPT-4o, Single-neuron activity, Interpretable representations, Naturalistic stimuli

TL;DR: SYNAPSE is a GPT-4o-powered framework that automatically extracts meaningful semantic features—like emotions, objects, and camera movements—from movie clips, without manual labeling.

Abstract: Understanding how the human brain represents rich, dynamic visual experiences requires feature spaces that are both interpretable and decodable from neural activity. Leveraging large language models to generate cross-modal embeddings, we present SYNAPSE (SYstematic Neural Analysis through Prompt-based Semantic Extraction), a GPT-4o-based framework for automatic extraction and categorization of semantically meaningful features from visual stimuli. SYNAPSE generates interpretable embeddings spanning emotions, actions, objects, and cinematographic features - everything from heart-racing tension and joyful laughter to dramatic close-ups and camera movements - all without manual labeling. We evaluate our approach by automatically extracting and then decoding features from single-neuron activity during an 8-minute movie clip, using four machine learning models: logistic regression, a multilayer perceptron, a long-short term memory network, and a bidirectional long-short term memory network. From comparison with manual labels, we find that our automatic labels accurately identify salient features with a false positive error rate under 1.5\%, and our results show that our framework is reliably able to determine what features are decodable across models, demonstrating the viability of our proposed framework in automatically discovering and generating interpretable representations of dynamic information aligned with neural activity. We release the codebase and our embeddings on https://anonymous.4open.science/r/synapse_anonymous_6/README.md

Submission Number: 59