## Measuring CLEVRness: Black-box Testing of Visual Reasoning Models
Spyridon Mouselinos, Henryk Michalewski, Mateusz Malinowski
Keywords: 
ICLR/2022/Proceedings/6011 - Measuring CLEVRness: Black-box Testing of Visual Reasoning Models.pdf
Project URL: nan

### Implementation
_Given the documentation shared by the authors on a new method, how much effort would it be to re-implement the method from scratch?_

[8]

The authors specify a few packages and source codes they used for their study in appendix B.2. An overview is given in figure 1. No other details given.

### Data
_Given the data description in the documentation, how much effort would it take to either: Find the same data set the authors used, or a similar data set and defend the comparability, or acquire one from scratch?_

[2]

(1/1)

The authors use the CLEVR dataset and cite and explain it in section 3 as well as how they use it for their experiment. Direct link missing.

### Configuration 
_Given the (hyper)parameters, including semantic parameters, of the method: How much effort would it take to acquire the algorithm configurations used for obtaining the reported results, and compare them against their computation budget?_

[10]

An architecture overview is given in figure 5. No mention of (hyper) parameter values in the paper.

### Experimental Procedure
_Given the setup of experiments reported in the work, how difficult is it to set up a new experiment with the same procedure, similar to those presented in the original work?_

[1]

The authors prsent the results with accuracy over the amount of games with each experiment being repeated 30 times. The setup for how the games are played is well explained in section 3 and how the CLEVR dataset is used for this.

### Expertise
_How much effort would it take to acquire the expertise required to reproduce the work independently relying solely on the available documentation?_

[6]

-
