Attention Highlights Help Humans Predict Model Behavior

Anonymous

TL;DR: Attention weights used to highlight parts of the text can help humans predict model output

Abstract: Although numerous studies have investigated whether or not attention can be used by \emph{researchers} as a tool of interpretability for understanding their models, a consensus has yet to be reached. This study aims to examine the attention mechanism practicality by testing whether attention can help the \emph{end-users} of such models to predict their behaviour by comparing their performance with and without access to attention highlights. We divided human evaluators into two groups—one with access to attention highlights and another without—to assess the performance differences between the two groups in terms of decision-making accuracy and response time. Our results showed that including attention highlights significantly improved decision-making accuracy for humans to predict extractive question-answering model output, with a notable difference in $F_1$ scores between the two groups. However, the time taken to predict model response was not significantly affected, suggesting that attention highlights did not speed up decision-making.

Paper Type: short

Research Area: Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability

Languages Studied: English