Which Neurons Matter in IR? Applying Integrated Gradients-based Methods to Understand Cross-Encoders

Mathias Vast, Basile Van Cooten, Laure Soulier, Benjamin Piwowarski

Keywords: Information Retrieval, Cross-Encoders, Interpretability, Integrated Gradients

TL;DR: With the help of the Neuron Integrated Gradients method, we explore the role of specific neurons in the IR task.

Abstract:

With the recent addition of Retrieval-Augmented Generation (RAG),
the scope and importance of Information Retrieval (IR) has expanded. As a result, the importance of a deeper understanding of
IR models also increases. However, interpretability in IR remains
under-explored, especially when it comes to the models’ inner
mechanisms. In this paper, we explore the possibility of adapting
Integrated Gradient-based methods in an IR context to identify the
role of individual neurons within the model. In particular, we provide new insights into the role of what we call "relevance" neurons,
as well as how they deal with unseen data. Finally, we carry out an
in-depth pruning study to validate our findings.

Submission Number: 40