Brain-to-4D: 4D Generation from fMRI

Yuankun Yang, Zijie Pan, Xiatian Zhu, Li Zhang

Keywords: diffusion, neuroscience, fMRI, Gaussian Splatting

TL;DR: We propose Brain-to-4D, a more powerful yet challenging BCI function to construct 4D visuals including both video and 3D directly from brain fMRI signals, and propose WSf4D, a novel Weakly Supervised decomposed fMRI-to-4D generation approach.

Abstract: Brain-computer interface (BCI) with functional magnetic resonance imaging (fMRI) has enabled new communication interfaces for many real-world applications, e.g., fMRI to image or video. While useful for specific scenarios (e.g., neurofeedback), the existing functions are limited in offering immersive user experience as required by more complex applications (e.g., virtual reality). We thus propose Brain-to-4D, a more powerful yet challenging BCI function to construct 4D visuals including both video and 3D directly from brain fMRI signals. In reality, however, it is infeasible to acquire brain signals for multi-view 4D stimuli for training data collection due to the instantaneity nature of brain activities. Typically, brain fMRI data exhibit significantly large variation. To address both obstacles, we introduce WSf4D, a novel Weakly Supervised decomposed fMRI-to-4D generation approach, characterized by foreground-background decomposition for supervision dividing and fMRI multifaceted vector quantization for noise suppression. To explore the application of the new task Brain-to-4D and our solution WSf4D, we conduct analysis and diagnosis on various brain regions by encoding distinct visual cortex groups. Extensive experiments show that WSf4D can accurately generate multi-view consistent 4D scenes semantically aligned with raw brain signals, indicating meaningful advancements over existing approaches on the potentials of neuroscience and diagnosis.

Primary Area: applications to neuroscience & cognitive science

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2016