Diversity-aware Evaluation for Paraphrase Patterns

Hideki Shima, Teruko Mitamura

Abstract: Common evaluation metrics for paraphrase patterns do not necessarily correlate with extrinsic recognition task performance. We propose a metric which gives weight to lexical variety in paraphrase patterns; our proposed metric has a positive correlation with paraphrase recognition task performance, with a Pearson correlation of 0.5~0.7 (k=10, with "strict" judgment) in a statistically significant level (p-value<0.01).