D3: Data Diversity Design for Systematic Generalization in Visual Question Answering
Abstract: Systematic generalization is a crucial aspect of intelligence, which refers to the ability to generalize to novel tasks by combining known subtasks and concepts. One critical factor that has been shown to influence systematic generalization is the diversity of training data. However, diversity can be defined in various ways, as data have many factors of variation. A more granular understanding of how different aspects of data diversity affect systematic generalization is lacking. We present new evidence in the problem of Visual Question Answering (VQA) that reveals that the diversity of simple tasks (i.e. tasks formed by a few subtasks and concepts) plays a key role in achieving systematic generalization. This implies that it may not be essential to gather a large and varied number of complex tasks, which could be costly to obtain. We demonstrate that this result is independent of the similarity between the training and testing data and applies to well-known families of neural network architectures for VQA (i.e. monolithic architectures and neural module networks). Additionally, we observe that neural module networks leverage all forms of data diversity we evaluated, while monolithic architectures require more extensive amounts of data to do so. These findings provide a first step towards understanding the interactions between data diversity design, neural network architectures, and systematic generalization capabilities.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission: We have added Qualitative results (Figure 4) and their description at the end of Section 3.1. Additional experiments on new datasets (TallyQA, GQA) and models (GPT-2+vision encoder, MiniGPT-v2) are conducted in Section 3.2. Additional results are presented in Tables 2,3,4,5,6. We have added a discussion and results about NS-VQA and ViperGPT in the Appendix F, and Table 9.
Assigned Action Editor: ~Neil_Houlsby1
Submission Number: 1923
Loading