A Benchmark of Variance of Opinion Scores in Image Quality Assessment

Jianxun Lou, Xinbo Wu, Yingying Wu, Padraig Corcoran, Gualtiero Colombo, Roger M. Whitaker, Hantao Liu

Abstract: Mean opinion score (MOS) has been used as the benchmark to measure the perceived quality of digital images. However, the usefulness of MOS diminishes when a substantial variation between individual opinions occurs. It is critical to measure the stimulus-driven variance of opinion scores (VOS) and scrutinise images that evoke a large VOS, and consequently, use VOS to inform our interpretation of MOS. In this paper, we create a VOS benchmark for individual differences in image quality assessment and analyse the importance of VOS classification as a function of distortion intensity, distortion type and scene content. In addition, a simple yet effective deep learning-based model is built, aiming to identify images with a large variation in viewers’ quality judgements.