Color is a Third Wheel in Shape-Texture Bias

Vatsala Nema, Aditya Pola, Vineeth N. Balasubramanian

Keywords: CV, Interpretability, Explainability and Transparency

TL;DR: Does color tilt the scale towards shape or texture

Abstract:

When faced with conflicting shape and texture cues, it is well established that con-
temporary neural networks rely on shape cues less often than humans do. Measur-
ing this gap in preference of shape cues allows us to move towards networks with
an increased shape bias. Complementary to previous works, our work posits that
there is an unexplored yet crucial confounding factor in this debate: color. We
hypothesize that color affects how conflicts between shape and texture cues are
resolved. We test our hypothesis across two dimensions, namely, color variations
and model architectures. To test our hypothesis on an increasing scale of color
variation, we propose color-variants with the following characteristics: a) only
magnitude is preserved, b) discrete colors, c) continuous variation in the color
space, and d) adversarial color variation. To test our hypothesis across the second
dimension, we perform our analysis on two model classes, namely, ViT and CLIP.
This necessitates a new way of measuring shape bias for which we propose a new
metric: Shape Precedence. Our findings extend to the varying nature of shape
bias across model architectures and color distortions at both a global and a local
(patch) level.

Primary Area: interpretability and explainable AI

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 12888