How to Design Sample and Computationally Efficient VQA Models

Karan Samel, Zelin Zhao, Kuan Wang, Robin Luo, Binghong Chen, Le Song

Keywords: vqa, visual question answering, neural modules, probabilistic logic

Abstract: In multi-modal reasoning tasks, such as visual question answering (VQA), there have been many modeling and training paradigms tested. Previous models propose different methods for the vision and language tasks, but which ones perform the best while being sample and computationally efficient? Based on our experiments, we find that representing the text as probabilistic programs and images as object-level scene graphs best satisfy these desiderata. We extend existing models to leverage these soft programs and scene graphs to train on question answer pairs in an end-to-end manner. Empirical results demonstrate that this differentiable end-to-end program executor is able to maintain state-of-the-art accuracy while being sample and computationally efficient.

One-sentence Summary: An exploration of which individual components of VQA methods are most efficient, and show that the combination of all these components provides the most efficient model.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=04g3mBawvt