A Systematic Comparison of fMRI-to-video Reconstruction Techniques

Camilo Luciano Fosco, Ben Lahner, Alex J Andonian, Bowen Pan, Aude Oliva

Keywords: video reconstruction, fMRI signals, fMRI reconstruction

TL;DR: We propose a new technique to generate watched videos from corresponding fMRI signals. We compare several different video generators under this paradigm and analyze their performance and particularities.

Abstract: Recent advances in generative models and large-scale neural datasets have brought forth novel methods to reconstruct stimuli from brain activity. This rapidly evolving family of brain-to-stimuli reconstruction techniques has the opportunity to revolutionize fundamental brain sciences and human-computer interaction applications, yet systemic comparisons of these techniques are lacking. Here, we explore a novel method to reconstruct short videos from functional magnetic resonance imaging (fMRI) brain activity of human subjects that achieves state-of-the-art performance as assessed by a suite of evaluation metrics. We perform preliminary comparisons of reconstruction quality within our pipeline by testing different combinations of semantic encoders and video generation models. Lastly, we compare our pipeline's best reconstruction results with previous work. Together, this work comprehensively assesses state-of-the-art methodologies in the increasingly important discipline of brain-to-video reconstruction.

Submission Number: 10