A Unified View of Evaluation Metrics for Structured Prediction

Yunmo Chen, William Gantt, Tongfei Chen, Aaron Steven White, Benjamin Van Durme

Submission Type: Regular Long Paper

Submission Track: Information Extraction

Submission Track 2: Resources and Evaluation

Keywords: structured prediction, information extraction, evaluation metrics, template extraction, n-ary relation extraction

Abstract: We present a conceptual framework that unifies a variety of evaluation metrics for different structured prediction tasks (e.g. event and relation extraction, syntactic and semantic parsing). Our framework requires representing the outputs of these tasks as objects of certain data types, and derives metrics through matching of common substructures, possibly followed by normalization. We demonstrate how commonly used metrics for a number of tasks can be succinctly expressed by this framework, and show that new metrics can be naturally derived in a bottom-up way based on an output structure. We release a library that enables this derivation to create new metrics. Finally, we consider how specific characteristics of tasks motivate metric design decisions, and suggest possible modifications to existing metrics in line with those motivations.

Submission Number: 935