Ground-truthing perspectives on highly subjective text: basic human values perceived in song lyrics

Anonymous

TL;DR: We collected a perspectivist and psychology-informed annotated dataset of human values in song lyrics, and describe it in detail.

Abstract: We present an interdisciplinary approach to creating a dataset on a highly subjective text annotation task. The task thus requires explicit insight into broader human annotator perspectives and perceptions, and conscious curation of what will be annotated. In this, with strong inspiration from best practices in the social sciences, we add to emerging and increasing calls for greater accountability with regard to data and its quality. For our task, we choose the annotation of perceived human values in song lyrics. Drawing from a representative US population sample, we present our strategy to select song lyrics to be annotated, estimate the amount of annotators needed, and assess data quality. Based on this, we obtain a dataset of 360 richly annotated song lyrics. We substantiate the benefit of having more annotators, and show how annotations show promising consistency with earlier insights on personal value proximity from a validated cross-cultural instrument study. Finally, we give a first illustration of how our data can be employed in connection to applied machine learning approaches.

Paper Type: long

Research Area: Linguistic theories, Cognitive Modeling and Psycholinguistics

Contribution Types: Data resources, Data analysis

Languages Studied: English