Ground-truthing perspectives on highly subjective text: basic human values perceived in song lyrics

Anonymous

Abstract: We present an interdisciplinary approach to gathering a dataset on a highly subjective text annotation task. The task thus requires explicit insight into broad human annotator perceptions, and conscious curation of what will be annotated. With strong inspiration from best practices in the social sciences, we add to emerging and increasing calls for greater accountability with regard to data and its quality. For our task, we choose the annotation of human values as they are perceived in song lyrics. We present our strategy to select song lyrics for annotation, draw annotators from a representative US sample, estimate number of annotators needed, and assess data quality. We obtain a dataset of 360 richly annotated lyrics, and substantiate the benefits of our approach, which can be adapted to many domains and tasks. Finally, we give a first illustration of how our data can be employed in connection to applied machine learning approaches.

Paper Type: long

Research Area: Computational Social Science and Cultural Analytics

Contribution Types: Data resources, Data analysis

Languages Studied: English