VALUED - Vision and Logical Understanding Evaluation Dataset

Soumadeep Saha, Saptarshi Saha, Utpal Garain

Abstract: Starting with early successes in computer vision tasks, deep learning based techniques have since overtaken state of the art approaches in a multitude of domains. However, it has been demonstrated time and again that these techniques fail to capture semantic context and logical constraints, instead often relying on spurious correlations to arrive at the answer. Since application of deep learning techniques to critical scenarios are dependent on adherence to domain specific constraints, several attempts have been made to address this issue. One limitation holding back a thorough exploration of this area, is a lack of suitable datasets which feature a rich set of rules. In order to address this, we present the VALUE (Vision And Logical Understanding Evaluation) Dataset, consisting of 200,000$+$ annotated images and an associated rule set, based on the popular board game - chess. The curated rule set considerably constrains the set of allowable predictions, and are designed to probe key semantic abilities like localization and enumeration. Alongside standard metrics, additional metrics to measure performance with regards to logical consistency is presented. We analyze several popular and state of the art vision models on this task, and show that, although their performance on standard metrics are laudable, they produce a plethora of incoherent results, indicating that this dataset presents a significant challenge for future works.

Keywords: logical constraints, domain knowledge, deep learning, computer vision

Previous DMLR Submission Url: https://openreview.net/forum?id=nS9oxKyy9u&noteId=nS9oxKyy9u

Changes Since Last Submission:

Added link to the [website](https://espressovi.github.io/VALUED) where demonstrations of the dataset samples are provided (Footnote on Page 3). Journal name and dates are added at the top in keeping with DMLR camera ready paper styles.

**No content has been changed.**

Video: https://youtu.be/6V9VlTEfHT4

Code: https://github.com/espressoVi/VALUE-Dataset

Assigned Action Editor: ~Sergio_Escalera1

Submission Number: 31