Predicting Document Relevance from Brain Recordings

Vadym Gryshchuk, Maria Maistro, Christina Lioma, Tuukka Ruotsalo

Abstract: Recent neuroimaging studies have revealed the association between relevance and brain responses. However, fundamental questions about how the human brain responds to a human relevance judgement of an entire text document and how such responses could be used in predicting document relevance remain unexplored. Here, we present the first work to utilise electroencephalography (EEG) data for predicting document relevance with respect to the topic selected by a human whose brain responses are recorded during document reading. Our approach jointly learns to predict document relevance from EEG and word embeddings computed for the document under a bimodal architecture. The EEG representations in our bimodal architecture account for a human’s attention towards words, and word embeddings are used as a representation of word semantics. Experiments with several EEG decoding models and word embedding models show that document relevance can be predicted from EEG data and that our bimodal approach yields higher prediction performance (AUROC=0.68) than models with only word embeddings (AUROC=0.62) or only EEG data (AUROC=0.63). Our findings create new opportunities for modelling document relevance through implicit physiological signals, emphasising the combined importance of human brain signals and language models in capturing personalised document relevance beyond traditional behavioural signals.