Non-Expert Evaluation of Summarization Systems is Risky
Abstract: We provide evidence that intrinsic evaluation of summaries using Amazon's Mechanical Turk is quite difficult. Experiments mirroring evaluation at the Text Analysis Conference's summarization track show that non-expert judges are not able to recover system rankings derived from experts.
0 Replies
Loading