Focused Evaluation for Image Description with Binary Forced-Choice Tasks

Micah Hodosh, Julia Hockenmaier

Abstract: Current evaluation metrics for image description may be too coarse. We therefore propose a series of binary forced-choice tasks that each focus on a different aspect of the captions. We evaluate a number of different off-the-shelf image description systems. Our results indicate strengths and shortcomings of both generation and ranking based approaches.