Estimating Agreement by Chance for Sequence Annotation

Anonymous

Abstract: The Kappa statistic is a popular chance corrected measure of agreement used as a reliability measure for annotation in the field of NLP, however its method for estimating chance agreement is not suitable for sequence annotation tasks, which are extremely prevalent in the field. The non-suitability is grounded in several complicating factors such as variation in span density across documents and constraints on span connectivity and overlap. In this paper, we propose a novel model for random annotation generation as the basis for chance agreement estimation for sequence annotation tasks. The model is jointly motivated by the specific characteristics of text sequence labeling tasks and acknowledgement of differences in annotation tendencies among annotators. Based on the proposed randomization model and related comparison approach, we successfully derive the analytical form of the distribution for computing the probable location of each annotated text segment, and subsequently chance agreement. We illustrate the approach in a simulation experiment and then apply it to several system outputs of CoNLL03 corpus annotation to evaluate its applicability, thus substantiating both the accuracy and efficacy of our method.

Paper Type: long

Research Area: Resources and Evaluation