Ranking Human and Machine Summarization Systems

Peter A. Rankel, John M. Conroy, Eric Slud, Dianne P. O'Leary

Abstract: The Text Analysis Conference (TAC) ranks summarization systems by their average score over a collection of document sets. We investigate the statistical appropriateness of this score and propose an alternative that better distinguishes between human and machine evaluation systems.