HYPE-C: Evaluating Image Completion Models Through Standardized Crowdsourcing

Emily Walters, Weifeng Chen, Jia Deng

Keywords: evaluation methods, image completion, image inpainting, evaluation, generative adversarial model, GAN, autoregressive generative model

Abstract: A significant obstacle to the development of new image completion models is the lack of a standardized evaluation metric that reflects human judgement. Recent work has proposed the use of human evaluation for image synthesis models, allowing for a reliable method to evaluate the visual quality of generated images. However, there does not yet exist a standardized human evaluation protocol for image completion. In this work, we propose such a protocol. We also provide experimental results of our evaluation method applied to many of the current state-of-the-art generative image models and compare these results to various automated metrics. Our evaluation yields a number of interesting findings. Notably, GAN-based image completion models are outperformed by autoregressive approaches.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We propose a standardized method to evaluate image completion models using human evaluators.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=JOfjm2ksZ