A Test Suite for Evaluating POS Taggers across Varieties of English

Anna Jørgensen, Anders Søgaard

Abstract: We present a suite of 12 datasets for evaluating POS taggers across varieties of English to enable researchers to evaluate the robustness of their models. The suite includes three new datasets, sampled from lyrics from black American hip-hop artists, southeastern American Twitter, and the subtitles from the TV series The Wire. We present an example eval- uation of an off-the-shelf POS tagger across these datasets.