BraiNav: Incorporating Human Brain Activity to Enhance Robustness in Embodied Visual Navigation
Keywords: embodied visual navigation, neural encoding, multimodal learning
Abstract: Recent research shows that standard navigation agents significantly underperform and even fail in the presence of various visual corruptions. Unlike embodied agents, the human brain's visual system can robustly perceive the environment and extract the necessary information to complete the visual tasks. In this paper, we propose a two-phase Brain-Machine integration Navigation method called BraiNav, which incorporates neural representations derived from human brain activity to enhance robustness against visual corruptions. In the first phase, a brain encoder, built upon a recently advanced self-supervised pretrained model, is trained on a large-scale human brain activity dataset and then frozen for downstream visual navigation. In the second phase, neural representations harboring high-level cognitive information from the human brain are constructed based on the pretrained frozen brain encoder. Additionally, we propose a multimodal fusion method based on cross-attention to obtain more consistent brain-visual joint representations, which are then used to learn the navigation policy. Sufficient experiments demonstrate that the proposed method exhibits higher robustness against various visual corruptions compared to standard navigation agent and multiple computer vision-enhanced agents. Our study pioneers the incorporation of human brain activity into embodied AI, aiming to catalyze further cross-disciplinary collaboration with computational neuroscience.
Primary Area: applications to neuroscience & cognitive science
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5700
Loading