Identifying and Measuring Token-Level Sentiment Bias in Pre-trained Language Models with Prompts

Anonymous

Paper Link: https://openreview.net/forum?id=Z1LmYdQluQz

Paper Type: Short paper (up to four pages of content + unlimited references and appendices)

Abstract: Due to the superior performance, large-scale pre-trained language models (PLMs) have been widely adopted in many aspects of human society. However, we still lack effective tools to understand the potential bias embedded in the black-box models. Recent advances in prompt tuning show the possibility to explore the internal mechanism of the PLMs. In this work, we propose two token-level sentiment tests: Sentiment Association Test (SAT) and Sentiment Shift Test (SST) which utilize the prompt as a probe to detect the latent bias in the PLMs. Our experiments on the collection of sentiment datasets show that both SAT and SST can identify sentiment bias in PLMs and SST is able to quantify the bias. The results also prove that fine-tuning can augment existing bias in PLMs.