Deep learning approaches for neural decoding across architectures andrecording modalities

Jesse A. Livezey, Joshua I. Glaser

Abstract: Decoding behavior, perception, or cognitive state directly from neural signals is critical for brain-computerinterface research and an import tool for systems neuroscience. In the last decade, deep learning has becomethe state-of-the-art method in many machine learning tasks ranging from speech recognition to image segmen-tation. The success of deep networks in other domains has led to a new wave of applications in neuroscience.In this article, we review deep learning approaches to neural decoding. We describe the architectures used forextracting useful features from neural recording modalities ranging from spikes to fMRI. Furthermore, we ex-plore how deep learning has been leveraged to predict common outputs including movement, speech, and vision,with a focus on how pretrained deep networks can be incorporated as priors for complex decoding targets likeacoustic speech or images. Deep learning has been shown to be a useful tool for improving the accuracy andflexibility of neural decoding across a wide range of tasks, and we point out areas for future scientific development.