Towards Multi-Perspective NLP Systems: A Thesis Proposal

Benedetta Muscato

Keywords: NLP, Responsible AI, Perspectivism, Human Annotations, Subjective tasks, Human Disagreement, Ethics in AI

TL;DR: This thesis proposes a framework for leveraging human disagreement as a valuable signal in subjective NLP tasks to build more Inclusive and Perspective-Aware AI systems.

Abstract:

In the field of Natural Language Processing (NLP), a common approach for resolving human disagreement involves establishing a consensus among multiple annotators. However, previous research shows that overlooking individual opinions can result in the marginalization of minority perspectives, particularly in subjective tasks, where annotators may systematically disagree due to their personal preferences. Emerging \textit{Multi-Perspective} approaches challenge traditional methodologies that treat disagreement as mere noise, instead recognizing it as a valuable source of knowledge shaped by annotators' diverse backgrounds, life experiences, and values.
This thesis proposal aims to (1) identify the challenges of designing disaggregated datasets i.e., preserving individual labels in human-annotated datasets for subjective tasks (2) propose solutions for developing Perspective-Aware by design systems and (3) explore the correlation between human disagreement and model uncertainty leveraging eXplainable AI techniques (XAI).
Our long-term goal is to create a framework adaptable to various subjective NLP tasks to promote the development of more responsible and inclusive models.

Archival Status: Archival

Paper Length: Long Paper (up to 8 pages of content)

Submission Number: 96