SI-Score: An image dataset for fine-grained analysis of robustness to object location, rotation and size

Jessica Yung, Rob Romijnders, Alexander Kolesnikov, Lucas Beyer, Josip Djolonga, Neil Houlsby, Sylvain Gelly, Mario Lucic, Xiaohua Zhai

Keywords: machine learning, robustness, computer vision, image classification, synthetic, synthetic data

Abstract: Before deploying machine learning models it is critical to assess their robustness. In the context of deep neural networks for image understanding, changing the object location, rotation and size may affect the predictions in non-trivial ways. In this work we perform a fine-grained analysis of robustness with respect to these factors of variation using SI-Score, a synthetic dataset. In particular, we investigate ResNets, Vision Transformers and CLIP, and identify interesting qualitative differences between these.

Submission Type: Full submission (technical report + code/data)

Co Submission: No I am not submitting to the dataset and benchmark track and will complete my submission by June 3.

TL;DR: A synthetic image dataset for fine-grained analysis of robustness to object location, rotation and size. We find insights on ResNet and ViT models.