Non-Expert Evaluation of Summarization Systems is Risky

Dan Gillick, Yang Liu

Abstract: We provide evidence that intrinsic evaluation of summaries using Amazon's Mechanical Turk is quite difficult. Experiments mirroring evaluation at the Text Analysis Conference's summarization track show that non-expert judges are not able to recover system rankings derived from experts.