Evaluating Sentence Compression: Pitfalls and Suggested Remedies

Courtney Napoles, Benjamin Van Durme, Chris Callison-Burch

Abstract: This work surveys existing evaluation methodologies for the task of sentence compression, identifies their shortcomings, and proposes alternatives. In particular, we examine the problems of evaluating paraphrastic compression and comparing the output of different models. We demonstrate that compression rate is a strong predictor of compression quality and that perceived improvement over other models is often a side effect of producing longer output.