Labels have Human Values: Value Calibration of Subjective Tasks

ACL ARR 2026 January Submission7825 Authors

Keywords: Pluralistic Value Alignment, Human Values in NLP, Annotation Subjectivity, Value Calibration, Subjective NLP

Abstract:

Building NLP systems for subjective tasks requires one to ensure their alignment to contrasting human values.
We propose the MultiCalibrated Subjective Task Learner framework (MC-STL), which clusters annotations into identifiable human value clusters by three approaches (similarity of annotator rationales, expert-value taxonomies or rater's sociocultural descriptors) and calibrates predictions for each value cluster by learning cluster-specific embeddings. We demonstrate MC-STL on several subjective learning settings, including ordinal, binary, and preference learning predictions, and evaluate it on multiple datasets covering toxic chatbot conversations, offensive social media posts, and human preference alignment.
The results show that MC-STL consistently outperforms the baselines that ignore the latent value structure of the annotations, delivering gains in discrimination, value-specific calibration, and disagreement-aware metrics.

Paper Type: Long

Research Area: Human-AI Interaction/Cooperation and Human-Centric NLP

Research Area Keywords: human-centered evaluation, value-centered design, human factors in NLP

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 7825