Joint Training Does Not Transfer Information between EEG and Image Classifiers

Hari Bharadwaj, Ronnie Bring Wilbur, Jeffrey Mark Siskind

Keywords: object classification, EEG, human vision, neuroscience, neuroimaging, brain-computer interface

TL;DR: We refute central claims in a paper published in PAMI.

Abstract:

Caution is necessary with machine-learning methods, and especially
computer-vision methods, to support brain processing claims from
neuroimaging data. A recent paper (Palazzo et al. 2021) proposes (i) a
joint-training process that does not use class information and (ii) a
bidirectional transfer of (a) image information to an EEG classifier
and (b) brain-activity information to an image classifier, such that
the joint embedding includes the shared image and brain-activity
information. These claims cannot be maintained: the training process
is initialized with class information, and joint training with EEG
degrades rather than improves the performance of the image encoder.
Moreover, theoretical solutions exist that entail no transfer beyond
class information in the joint embedding space.

Primary Area: applications to neuroscience & cognitive science

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3645