Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)

Published: 14 Jan 2025, Last Modified: 14 Jan 2025Accepted by TMLREveryoneRevisionsBibTeXCC BY 4.0
Abstract: Can artificial intelligence unlock the secrets of the human brain? How do the inner mechanisms of deep learning models relate to our neural circuits? Is it possible to enhance AI by tapping into the power of brain recordings? These captivating questions lie at the heart of an emerging field at the intersection of neuroscience and artificial intelligence. Our survey dives into this exciting domain, focusing on human brain recording studies and cutting-edge cognitive neuroscience datasets that capture brain activity during natural language processing, visual perception, and auditory experiences. We explore two fundamental approaches: encoding models, which attempt to generate brain activity patterns from sensory inputs; and decoding models, which aim to reconstruct our thoughts and perceptions from neural signals. These techniques not only promise breakthroughs in neurological diagnostics and brain-computer interfaces but also offer a window into the very nature of cognition. In this survey, we first discuss popular representations of language, vision, and speech stimuli, and present a summary of neuroscience datasets. We then review how the recent advances in deep learning transformed this field, by investigating the popular deep learning based encoding and decoding architectures, noting their benefits and limitations across different sensory modalities. From text to images, speech to videos, we investigate how these models capture the brain's response to our complex, multimodal world. While our primary focus is on human studies, we also highlight the crucial role of animal models in advancing our understanding of neural mechanisms. Throughout, we mention the ethical implications of these powerful technologies, addressing concerns about privacy and cognitive liberty. We conclude with a summary and discussion of future trends in this rapidly evolving field. Given the large amount of recently published work in the computational cognitive neuroscience (CCN) community, we believe that this survey provides an invaluable entry point for deep neural network (DNN) researchers looking to diversify into CCN research, inviting them to join in unraveling the ultimate puzzle: the human brain.
Certifications: Survey Certification
Submission Length: Long submission (more than 12 pages of main content)
Previous TMLR Submission Url: https://openreview.net/forum?id=nEFjCCCMyH
Changes Since Last Submission: - All reviewers' concerns have been thoroughly addressed in the final camera-ready version. - Based on the AE's suggestion, we have included a paragraph in the final camera-ready version that provides a high-level description of research topics related to brain-machine alignment in non-human subjects.
Code: https://github.com/subbareddy248/Awesome-Brain-Encoding--Decoding
Assigned Action Editor: ~Simon_Kornblith1
Submission Number: 2993
Loading