Multi-View Knowledge Distillation from Crowd Annotations for Out-of-Domain Generalization

Anonymous

Abstract: Selecting an effective training signal for tasks in natural language processing is difficult: expert annotations are expensive, and crowd-sourced annotations may not be reliable. At the same time, recent work in NLP has demonstrated that learning from a distribution over labels acquired from crowd annotations can be effective. However, the best method for acquiring these soft labels is inconsistent across tasks. This paper systematically analyzes this in the out-of-domain setting, adding to the NLP literature which has focused on in-domain evaluation, and proposes new methods for acquiring soft-labels from crowd-annotations by aggregating the distributions produced by existing methods. In particular, we propose to aggregate multiple-views of crowd annotations via temperature scaling and finding their Jensen-Shannon centroid. We demonstrate that these aggregation methods lead to best or near-best performance across four NLP tasks on out-of-domain test sets, mitigating fluctuations in performance when using the individual distributions. Additionally, aggregation results in best or near-best uncertainty estimation. We argue that aggregating different views of crowd-annotations is an effective way to ensure performance which is as good or better than the best individual view, which is useful given the inconsistency in performance of the individual methods.

Paper Type: long

Research Area: Machine Learning for NLP