Rethinking Brain-to-Image Reconstruction: What Should We Decode from fMRI Signals?

Jing Wang, Xiaoyu Han, Jinhui Tang, Shengping Zhang, Xiangyang Ji

Keywords: neural decoding, brain-to-image reconstruction, pseudo-foveated image synthesis, fMRI-to-foveated image decoding

Abstract:

Recently, notable advancements have been achieved in brain-to-image reconstruction. However, the assumption that the recorded brain activities faithfully mirror the complete high-resolution images conflicts with the workings of human vision and cognitive systems.
In this study, we present a novel approach, fMRI-to-foveated image (FitFovea), which redefines the brain-to-image reconstruction process to better align with cognitive science principles. FitFovea comprises three key stages: pseudo-foveated image synthesis, fMRI-to-foveated image reconstruction and stimulus image generation. In the first stage, FitFovea constructs new {fMRI, pseudo-foveated image} pairs from existing fMRI-image data using saliency prediction and foveated rendering techniques. Next, during the foveated image reconstruction phase, the information captured by human vision is decoded from fMRI signals with maximum accuracy. The final stage, stimulus image generation, is considered not as a strict reconstruction but rather as a postprocessing step. This stage is akin to existing brain-to-image decoding methods, which often emphasize semantic fidelity rather than pixel-level reconstruction. To validate our approach, we introduce the brain score metric to quantify the correlation between images and corresponding brain responses. The superior results validate the rationale behind decoding pseudo-foveated images from fMRI data and demonstrate the feasibility of our newly-devised pipeline based on synthesized pseudo-foveated image training data.

Primary Area: applications to neuroscience & cognitive science

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2141