Coding Varied Behavior Types Using the Crowd

Jinyeong Yim, Jeel Jasani, Aubrey Henderson, Danai Koutra, Steven Dow, Winnie Leung, Ellen Lim, Mitchell L. Gordon, Jeffrey P. Bigham, Walter S. Lasecki

Abstract: Social science researchers spend significant time annotating behavioral events in video data in order to quantitatively assess interactions [2]. These behavioral events may be instantaneous changes, continuous actions that span unbounded periods of time, or behaviors that would be best described by severity or other scalar ratings. The complexity of these judgments, coupled with the time and effort required to meticulously assess video, results in a training and evaluation process that can take days or weeks. Computational analysis of video data is still limited due to the challenges introduced by objective interpretation and varied contexts. Glance [4] introduced a means of leveraging human intelligence by recruiting crowds of paid online workers to accurately analyze hours of video data in a matter of minutes. This approach has been shown to expedite work in human-centered fields, as well as generate training data for automated recognition systems. In this paper, we describe an interactive demonstration of an improved, more expressive version of Glance that expands the initial set of supported annotation formats (e.g. time range, classification, etc.) from one to nine. Worker interfaces for each of these options are dynamically generated, along with tutorials, based on the analyst's question. These new features allow analysts to acquire more specific information about events in video datasets.