Benchmarking 360° Saliency Models by General-Purpose Metrics

Guanchen Li, Xiangjie Sui, Jiebin Yan, Yuming Fang

Abstract: How to effectively evaluate a model's capability to predict the visual attention of observers in 360° scenes gains interest along with the advancement of saliency prediction modeling of omnidirectional images (ODIs). So far, many general-purpose metrics from 2D saliency literature have been adopted to evaluate the 360° saliency models. However, whether they are still effective when being adopted to evaluate the 360° saliency models has not been explored. In this paper, we testify several standard saliency evaluation metrics on the 360° saliency models and comprehensively analyze their behaviors in the omnidirectional scenario. We find that 1) most metrics under-penalize false positives; 2) existing 360° datasets involve severe equator bias that few metrics can effectively penalize. We hope this case study can provide a guideline for benchmarking 360° image/video saliency models.