MonkeySee: Space-time-resolved reconstructions of natural images from macaque multi-unit activity

Lynn Le, Paolo Papale, K. Seeliger, Antonio Lozano, Thirza Dado, Feng Wang, Pieter R. Roelfsema, Marcel van Gerven, Yağmur Güçlütürk, Umut Güçlü

Keywords: space-time dependent reconstructions, biological constraints, neural decoding, deep neural network, receptive fields, visual cortex, multi unit activity, convolutional model, brain image reconstruction, naturalistic stimuli

TL;DR: This study uses a CNN to reconstruct images from macaque brain signals and explores how different brain areas process these signals at different timepoints.

Abstract: In this paper, we reconstruct naturalistic images directly from macaque brain signals using a convolutional neural network (CNN) based decoder. We investigate the ability of this CNN-based decoding technique to differentiate among neuronal populations from areas V1, V4, and IT, revealing distinct readout characteristics for each. This research marks a progression from low-level to high-level brain signals, thereby enriching the existing framework for utilizing CNN-based decoders to decode brain activity. Our results demonstrate high-precision reconstructions of naturalistic images, highlighting the efficiency of CNN-based decoders in advancing our knowledge of how the brain's representations translate into pixels. Additionally, we present a novel space-time-resolved decoding technique, demonstrating how temporal resolution in decoding can advance our understanding of neural representations. Moreover, we introduce a learned receptive field layer that sheds light on the CNN-based model's data processing during training, enhancing understanding of its structure and interpretive capacity.

Primary Area: Neuroscience and cognitive science (neural coding, brain-computer interfaces)

Submission Number: 19955