Validated Image Caption Rating (VICR) Scale, Dataset, and Model

Anonymous

Abstract: Assessing the quality of an image caption is a complex task. We propose a new image caption rating system that consists of (1) a robust rating scale that is consistent, teachable, and externally validated, (2) an engaging and scalable data generation approach for the task, (3) a high-quality dataset, and (4) an effective image caption rating predictor. Using contemporary approaches from psychometrics we demonstrate that the proposed scale and rater training routine can support high quality annotation efforts for the task. We introduce two new datasets (one original and another derived) for the task. Our reference-free and multi-level rating predictor performance is on par with state-of-the-art approaches.

Paper Type: long