Perceived Image Reconstruction from Human Brain Activity via Time-Series Information Guided Generative Adversarial Networks

Shuo Huang, Liang Sun, Muhammad Yousefnezhad, Meiling Wang, Daoqiang Zhang

Abstract: Understanding how human brain works has attracted increasing attentions in both fields of neuroscience and machine learning. Previous studies have used autoencoder and generative adversarial networks (GAN) to improve the quality of perceived image reconstruction from functional Magnetic Resonance Imaging (fMRI) data. However, these methods mainly focus on acquiring relevant features between stimuli images and fMRI while ignoring the time-series information of fMRI, thus leading to sub-optimal performance. To address this issue, in this paper, we develop a time-series information guided GAN method for reconstructing visual stimuli from human brain activities. In addition, to better measure the modal difference, we leverage a pairwise ranking loss to rank the stimuli images and fMRI to ensure strongly associated pairs at the top and weakly related ones at the bottom. Experimental results on real-world datasets suggest that the proposed method achieves better performance in comparison with several state-of-the-art image reconstruction approaches.