Query Augmentation with Brain Signals

Ziyi Ye, Jingtao Zhan, Qingyao Ai, Yiqun LIU, Maarten de Rijke, Christina Lioma, Tuukka Ruotsalo

Abstract:

In the information retrieval scenario, query augmentation is an essential technique to refine semantically imprecise queries to align closely with users' actual information needs.
Traditional methods typically rely on extracting signals from user interactions such as browsing or clicking behaviors to augment the queries, which may not accurately reflect the actual user intent due to inherent noise and the dependency on initial user interactions.
To overcome these limitations, we introduce Brain-Aug, a novel approach that decodes semantic information directly from brain signals of users to augment query representation.
Brain-Aug explores three-fold techniques:
(1) Structurally, an adapter network is utilized to project brain signals into the embedding space of a language model, allowing query augmentation conditioned on both the users' initial query and their brain signals.
(2) During training, we use a next token prediction task for query augmentation and adopt prompt tuning to efficiently train the brain adapter.
(3) At the inference stage, a ranking-oriented decoding strategy is implemented, enabling Brain-Aug to generate augmentations that improve ranking performance.
We evaluate our approach on multiple functional magnetic resonance imaging (fMRI) datasets, demonstrating that Brain-Aug not only produces semantically richer queries but also significantly improves document ranking accuracy, particularly for ambiguous queries.
These results validate the effectiveness of our proposed Brain-Aug approach, and reveal the great potential of leveraging internal cognitive states to understand and augment text-based queries.

Primary Subject Area: [Engagement] Multimedia Search and Recommendation

Secondary Subject Area: [Engagement] Emotional and Social Signals

Relevance To Conference: This paper proposes a novel technique that enhances multimodal search performance by integrating semantic information from brain signals. Based on the multimodal prompt tuning method, this technique combines brain modalities and text modalities to enhance query representation, which is particularly effective especially when textual queries are ambiguous.

Submission Number: 5368