System Identification of Neural Systems: Going Beyond Images to Modelling Dynamics

Mai Gamal, Mohamed Rashad Abdel Hamid, Eman Ehab Nasef, Seif Eldawlatly, Mennatullah Siam

Primary Area: applications to neuroscience & cognitive science

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Neuroscience, Neural Encoding, Video Understanding

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Vast literature has compared the recordings of biological neurons in the brain to deep neural networks. The ultimate goal is either reporting insights to interpret deep networks or to have a better understanding and encoding of biological neural systems. Recently, there has been a debate on whether system identification is possible and how much it can tell us about the brain computation. System identification recognizes whether one model is more valid to represent the brain computation over another. Nonetheless, previous work did not consider the time aspect and how video and dynamics (e.g., motion) modelling in deep networks compare to these biological neural systems. Towards this end, we propose a system identification study focused on comparing single image versus video understanding models with respect to the visual cortex recordings. Our study encompasses two sets of experiments; a real environment setup (i.e., regressing on the output of the visual cortex in the human brain recorded as fMRI responses) and a simulated environment setup (i.e., regressing on another network architecture representations that we know its modelling scheme). This study encompasses more than 30 models and, unlike prior works, we focus on convolutional versus transformer-based, single versus two-stream, and fully versus self-supervised video understanding models. The goal is to capture a greater variety of architectures that model dynamics. As such, this signifies the first large-scale study of video understanding models from a neuroscience perspective. Our results in the simulated experiments, show that system identification can be attained to a certain level. Moreover, we present the results of the real experiments and provide key insights on how dynamics modelling in deep networks compare to the human visual cortex.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3872