Image-to-Brain Signal Generation for Visual Prosthesis with CLIP Guided Multimodal Diffusion Models

Ganxi Xu, Boyu Wang, Jia Zhang, Jian Zhu, Jinyi Long

Keywords: Visual prostheses, image-to-brain, CLIP, diffusion models, cross-attention

Abstract: Visual prostheses hold great promise for restoring vision in blind individuals. While researchers have successfully utilized M/EEG signals to evoke visual perceptions during the brain decoding stage of visual prostheses, the complementary process of converting images into M/EEG signals in the brain encoding stage remains largely unexplored, hindering the formation of a complete functional pipeline. In this work, we present, to our knowledge, the first image‑to‑brain signal framework that generates M/EEG from images by leveraging denoising diffusion probabilistic models enhanced with cross‑attention mechanisms. Specifically, the proposed framework comprises two key components: a pretrained CLIP visual encoder that extracts rich semantic representations from input images, and a cross‑attention enhanced U‑Net diffusion model that reconstructs brain signals through iterative denoising. Unlike conventional generative models that rely on simple concatenation for conditioning, our cross‑attention modules capture the complex interplay between visual features and brain signal representations, enabling fine‑grained alignment during generation. We evaluate the framework on two multimodal benchmark datasets and demonstrate that it generates biologically plausible brain signals. We also present visualizations of M/EEG topographies across all subjects in both datasets, providing intuitive demonstrations of intra‑subject and inter‑subject variations in brain signals.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 7110