LEA: Learning Latent Embedding Alignment Model for fMRI Decoding and Encoding
Abstract: The connection between brain activity and visual stimuli is crucial to understand the human brain. While deep generative models have exhibited advances in recovering brain recordings by generating images conditioned on fMRI signals, it is still challenge to generate consistent semantics. Moreover, the prediction of fMRI signal from visual stimuli remains a hard problem. In this paper, we introduce a unified framework that addresses both fMRI decoding and encoding. With training two latent spaces to represent and reconstruct fMRI signals and visual images, respectively, we align the fMRI signals and visual images within the latent spaces, allowing us to transform between the two seamlessly. Our model, called Latent Embedding Alignment (LEA), concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images. LEA outperforms existing methods on multiple benchmark fMRI decoding and encoding datasets. LEA offers a comprehensive solution for modeling the relationship between fMRI signal and visual stimuli.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Bertrand_Thirion1
Submission Number: 2365
Loading