MindGrapher: Dynamic-Aware fMRI-to-Video Reconstruction
Keywords: fMRI; fMRI-to-Video; Dynamic-Aware; Video Reconstruction from Brain Activities;
Abstract: Existing methods for fMRI-to-video reconstruction typically focus on accurately reconstructing visual content ($i.e.$, appearance), neglecting dynamic event information. However, as highlighted in cognitive neurology, these key dynamic events significantly influence brain signal changes during video perception. In this article, we introduce Mindgrapher, a two-stream framework designed to address this gap by enhancing the reconstruction of dynamic-aware videos from fMRI data. Mindgrapher comprises $i)$ a visual content reconstruction stream, that improves the accuracy of the reconstructed visual content from sparsely distributed fMRI data through a temporal dynamics enrichment approach and multi-moment multimodal contrastive learning; $ii)$ a dynamics injection stream, that firstly crafts dynamic-aware fMRI embeddings and then integrates them into the reconstruction process via a fine-grained approach, thereby producing videos that effectively perceive dynamic events. Moreover, to address the lack of suitable metrics for evaluating dynamic event information, we introduce a new evaluation metric named dynamic content fidelity (DCF), which measures how accurately dynamic events within the video are reconstructed. Upon evaluation with a publicly available fMRI dataset, Mindgrapher outperforms the state-of-the-arts on all metrics, $i.e.$, semantic classification accuracy, structural similarity index, and DCF. The reconstructed video results are available on our web page. Code shall be released.
Primary Area: applications to neuroscience & cognitive science
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3510
Loading