Neuro-Symbolic Decoding of Neural Activity

Yanchen Wang, Joy Hsu, Ehsan Adeli, Jiajun Wu

Keywords: neural decoding, concept grounding, neuro-symbolic learning

TL;DR: A neuro-symbolic framework for fMRI decoding with concept grounding.

Abstract: We propose NEURONA, a neuro-symbolic framework for fMRI decoding and concept grounding in neural activity. Leveraging image- and video-based fMRI question-answering datasets, NEURONA learns to decode interacting concepts from visual stimuli based on patterns of fMRI responses, integrating symbolic reasoning and compositional execution with fMRI grounding across brain regions. We demonstrate that incorporating structural priors (e.g., compositional predicate-argument dependencies between concepts) into the decoding process significantly improves both decoding accuracy over precise queries, and notably, generalization to unseen queries at test time. With NEURONA, we highlight neuro-symbolic frameworks as promising tools for understanding neural activity.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 14108