Go to ACL ARR 2025 February homepagePersEval: A framework for Perspectivist Classification Evaluation
Abstract: Natural Language Processing systems must be able to understand and adapt to diverse perspectives, distinguishing between varying viewpoints and subjective interpretations -- an approach often referred to as perspectivism. While previous research has highlighted the need for moving beyond a single gold standard in evaluation, current practices remain fragmented and do not fully capture the complexity of perspectivist classification. To address this gap, we introduce PersEval, the first unified framework for evaluating perspectivist models in NLP. A key innovation of this framework is its treatment of annotators and users as disjoint entities. This mirrors real-world scenarios where the individuals providing annotations to train models are distinct from the end-users whose perspectives the system must learn and accommodate. We instantiate PersEval by experimenting with several encoder-based and decoder-based approaches. The results consistently show improvements when the models are informed with knowledge about the users.
Paper Type: Long
Research Area: Resources and Evaluation
Research Area Keywords: benchmarking, language resources
Contribution Types: Model analysis & interpretability, Data resources
Languages Studied: English
Submission Number: 2441
Loading