A tale of three parsers: Towards diagnostic evaluation for meaning representation parsing

Maja Buljan, Joakim Nivre, Stephan Oepen, Lilja Øvrelid

Abstract:

We discuss methodological choices in contrastive and diagnostic evaluation in meaning representation parsing, i.e. mapping from natural
language utterances to graph-based encodings of semantic structure. Drawing inspiration from earlier work in syntactic dependency
parsing, we transfer and refine several quantitative diagnosis techniques for use in the context of the 2019 shared task on Meaning
Representation Parsing (MRP). As in parsing proper, moving evaluation from simple rooted trees to general graphs brings along its own
range of challenges. Specifically, we seek to begin to shed light on relative strenghts and weaknesses in different broad families of parsing
techniques. In addition to these theoretical reflections, we conduct a pilot experiment on a selection of top-performing MRP systems
and two of the five meaning representation frameworks in the shared task. Empirical results suggest that the proposed methodology can
be meaningfully applied to parsing into graph-structured target representations, uncovering hitherto unknown properties of the different
systems that can inform future development and cross-fertilization across approaches.