Annotating live messages on social media. Testing the efficiency of the AnnotHate – live data annotation portal

Gunther Jikeli, Katharina Soemer, Sameer Karali

Abstract: Labeling datasets to produce gold standard corpora for training machine learning algorithms a re increasingly important in social media research. The annotation process, including annotation tools, is of utmost importance to the quality of gold standard corpora. While measuring inter-annotator reliability has become standard practice, and research has been conducted on the annotators themselves and their possible influence on the annotation process, reflections on the annotation tools often remain neglected in descriptions of gold standard productions. Many social media posts are short and require more context to understand their meaning, which only the live environment can provide. However, most annotation tools work with offline data. We test a specially designed tool for live data annotation, including an experiment with 80 annotators. The tool is user-friendly for annotators, does not require any command line usage or installations, and reduces errors in the annotation process. It is time efficient in the annotation process, and efficient and transparent in collecting the data from the annotation.

External IDs:doi:10.1007/s42001-024-00251-0