What makes a good metric? Evaluating automatic metrics for text-to-image consistency

Candace Ross, Melissa Hall, Adriana Romero-Soriano, Adina Williams

Research Area: Evaluation, Science of LMs

Keywords: text-to-image, evaluation, automatic evaluation, multimodal, vision-language

TL;DR: We define desiderata for strong automatic evaluation metrics for text-to-image models and find existing metrics fail to fulfill them all and are susceptible to shortcuts.

Abstract: Language models are increasingly being incorporated as components in larger AI systems for various purposes, from prompt optimization to automatic evaluation. In this work, we analyze the construct validity of four recent, commonly used methods for measuring text-to-image consistency---CLIPScore, TIFA, VPEval, and DSG---which rely on language models and/or VQA models as components. We define construct validity for text-image consistency metrics as a set of desiderata that text-image consistency metrics should have, and find that no tested metric satisfies all of them. We find that metrics lack sufficient sensitivity to language and visual properties. Next, we find that TIFA, VPEval and DSG contribute novel information above and beyond CLIPScore, but also that they correlate highly with each other. We also ablate different aspects of the text-image consistency metrics and find that not all model components are strictly necessary, also a symptom of insufficient sensitivity to visual information. Finally, we show that all three VQA-based metrics likely rely on familiar text shortcuts (such as yes-bias in QA) that call their aptitude as quantitative evaluations of model performance into question.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html

Author Guide: I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html

Submission Number: 965