BinaryVQA: A Versatile Dataset to Push the Limits of VQA Models

ali borji

Keywords: Visual question answering, dataset benchmarks, datasets

TL;DR: We introduce a new test set for free-form visual question answering (VQA) called BinaryVQA to push the limits of VQA models.

Abstract:

We introduce a new test set for visual question answering (VQA) called BinaryVQA
to push the limits of VQA models. Our dataset includes 7,800 questions across
1,024 images and covers a wide variety of objects, topics, and concepts. For easy
model evaluation, we only consider binary questions. Questions and answers are
formulated and verified carefully and manually. Around 63% of the questions
have positive answers. The median number of questions per image and question
length are 7 and 5, respectively. The state of the art OFA model achieves 75%
accuracy on BinaryVQA dataset, which is significantly lower than its performance
on the VQA v2 test-dev dataset (94.7%). We also analyze the model behavior along
several dimensions including a) performance over different categories such as text,
counting and gaze direction, b) model interpretability, c) the effect of question
length on accuracy, d) bias of models towards positive answers and introduction of
a new score called the “ShuffleAcc”, and e) sensitivity to spelling and grammar
errors. Our investigation demonstrates the difficulty of our dataset and shows that it
can challenge VQA models for years to come. Data and code is available [Masked].

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Infrastructure (eg, datasets, competitions, implementations, libraries)