ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation

Xinpeng Wang, Barbara Plank

Submission Type: Regular Short Paper

Submission Track: Human-Centered NLP

Submission Track 2: Efficient Methods for NLP

Keywords: Active Learning, Human Label Variation, Multi-task Learning

TL;DR: In active learning, a model with annotator-specific heads improves performance while reducing the annotation budget by 70%, leveraging group and individual uncertainty based on dataset properties.

Abstract:

Label aggregation such as majority voting is commonly used to resolve annotator disagreement in dataset creation.
However, this may disregard minority values and opinions.
Recent studies indicate that learning from individual annotations outperforms learning from aggregated labels, though they require a considerable amount of annotation.
Active learning, as an annotation cost-saving strategy, has not been fully explored in the context of learning from disagreement.
We show that in the active learning setting, a multi-head model performs significantly better than a single-head model in terms of uncertainty estimation.
By designing and evaluating acquisition functions with annotator-specific heads on two datasets, we show that group-level entropy works generally well on both datasets.
Importantly, it achieves performance in terms of both prediction and uncertainty estimation comparable to full-scale training from disagreement, while saving 70\% of the annotation budget.

Submission Number: 3161