When Disagreement Meets Noise: Noise Robust Annotator Embeddings for Subjective NLP

ACL ARR 2025 May Submission6195 Authors

Abstract: Subjective NLP tasks such as sentiment analysis and hate speech classification often involve inherent annotator disagreement, reflecting diverse perspectives shaped by annotators’ lived experiences. Although conventional approaches resolve disagreement through majority voting or aggregation, these methods risk erasing valuable nuances and minority viewpoints. Recent embedding-based/multitask models have advanced the modeling of annotator-specific judgments, yet their robustness to annotation noise remains underexplored. In this work, we systematically investigate how state-of-the-art disagreement learning models perform in the presence of noisy labels and observe a significant performance degradation under such conditions. To address this, we propose Noise Robust Annotator Embedding (NRA-Embed), which integrates Robust InfoNCE (RINCE) contrastive loss to enhance models' robustness under noisy annotation conditions. Moreover, we benchmark existing approaches across three axes: label noise type (symmetric vs. rogue annotators), task structure (binary vs. multiclass), and annotator coverage (many vs. few labels per example). Through extensive experiments, we show that NRA-Embed effectively models subjective variation while remaining resilient to noise, achieving competitive or superior performance compared to prior methods.

Paper Type: Long

Research Area: Computational Social Science and Cultural Analytics

Research Area Keywords: hate-speech detection, stance detection, model bias/unfairness mitigation, ethical considerations in NLP applications

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 6195