Are Two Heads Better Than One? An Exploration of Ambiguity in Crowd-Collected Language Decisions from the Phrase Detectives Game

Jon Chamberlain

Abstract: The online game-with-a-purpose Phrase Detectives (https://www.phrasedetectives.com) [1] has been collecting decisions about anaphoric coreference in human language for over 10 years (4 million judgements from 40,000 players). The game was originally designed to collect multiple valid solutions for a single task, which complicated aggregation but created a very rich (and noisy) dataset [2]. Analysis of the ambiguous player decisions highlights the need for understanding and resolving disagreement that is inherent in language interpretation. This talk will present some of the interesting cases of ambiguity found by the players of Phrase Detectives (a dataset that will be made available to the research community later this year [3]) and discuss the statistical methods we have been working on to harness crowds that disagree with each other [4, 5].