MindDETR: Beyond Semantics, Exploring Positional Cues from Brain Activity

Han Deng, Xinzhu Ma, Yuan Meng, Chen Tang, Chunfeng Song, Zhi Wang, Wenwu Zhu, Xiangyu Yue, Wanli Ouyang

Keywords: brain decoding, object detection

TL;DR: We present a novel approach for object detection using brain signal data.

Abstract: Decoding visual stimuli from brain recordings offers a unique opportunity to understand how the brain represents the world and seeks to interpret the connection between computer vision models and our visual system. Recent efforts mainly adopt diffusion models to reconstruct images from brain signals. However, while these methods generally capture correct semantic information, they often struggle with precise object localization. Additionally, the commonly used proxy task, image reconstruction from brain signals, mainly measures semantic consistency, to some extent neglecting positional information of the decoded signals. In this work, to encourage more accurate brain signal decoding, we propose to use object detection as the proxy task, aiming at decoding both the semantic and positional cues from brain recordings. Based on this task, we propose MindDETR, a brain recording-based object detection model with the DETR pipeline. After aligning feature representations with a pretrained image-based DETR model, our model demonstrates that accurately brain decoding at both semantic and positional levels is feasible, and our detection-based approach achieves significantly superior results than existing reconstruction-based approaches. This result suggests the effectiveness of applying object detection as a proxy task for brain signal decoding. Our code will be publicly available.

Primary Area: applications to neuroscience & cognitive science

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6071